Engineering intelligent systems for industrial edge device fleets is often hindered by the non-functional requirements of data privacy, transmission cost, and client heterogeneity, which make traditional centralized architectures impractical. This paper presents a generalizable architectural blueprint designed to address these constraints for resource-constrained, time-series forecasting tasks. The blueprint is founded on a dual-adaptation mechanism that combines a longterm, asynchronous Federated Learning (FL) protocol with a lightweight, on-device correction mechanism for rapid, real-time adaptation. To validate this blueprint, we apply it to the challenging use case of real-time State of Charge (SoC) prediction for a heterogeneous fleet of commercial Battery Electric Vehicles (BEVs) and the NASA C-MAPSS dataset to evaluate Remaining Useful life (RUL) of jetengines. The system was implemented and evaluated on production-intent embedded ARM hardware. The results demonstrate that the blueprint enables a privacypreserving federated system to achieve a final prediction error nearly on par with a non-private, fully centralized model. In the SoC experiment, the client with the least amount of data, reduced the median end-of-trip error from over 4% in a local-only model to under 1.5% while the RUL experiment reduced the local model median error of 5.85 cycles down to 0.16 using the blueprint. This work provides a definitive blueprint for deploying effective, privacy-preserving AI systems in industrial embedded systems, proving that a thoughtful engineering approach allows performance and privacy to coexist without compromise.

Software-intensive systems companies face mounting pressures to accelerate time-to-market. This drives adoption of DevOps, frequent deployments and AI-enabled analytics. However, while traditional feedback loops channel usage data, logs and interactions into development cycles, this doesn’t necessarily translate into learning. Although companies use DevOps, they still view product development as building products with a fixed scope. To address this, we conceptualize the notion of learning loops and how companies move towards continuous improvement of product performance. In our view,’learning loops’ distinguish themselves by translating data from products into actionable improvements executed by humans, by traditional ML, by Agentic AI or by a combination of these. This paper synthesizes longitudinal case study research and interviews, revealing that mechanisms like continuous integration and deployment, A/B testing, federated and reinforcement learning are unified instances of post-deployment learning loops. The contribution of this paper is two-fold. First, we provide empirical examples reflecting how R&D teams and systems learn and improve performance over time. Second, we present a conceptual model in which we detail the concept of learning loops that can be executed by humans, by traditional ML, by Agentic AI or by a combination of these.

Within the Engineering, Procurement, and Construction (EPC) industry, engineers manually create documents based on engineering drawings, which can be time-consuming and prone to human error. For example, the expansion of typical assemblies of instrument items (Instrument Typicals) in Piping and Instrumentation Diagrams (P&IDs) is a labor-intensive task. Each Instrument Typical assembly is depicted in the P&IDs via a simplified representation showing only a subset of the utilized instruments. The expansion activity involves recording all utilized instruments to create an instrument item list document based on the P&IDs for a particular EPC project. Fortunately, Artificial Intelligence (AI) could help to automate this process. In this paper, we propose the first method for automating the process of Instrument Typical expansion in P&IDs. The method utilizes computer vision techniques and domain knowledge rules to extract information about the Instrument Typicals from a project's P&IDs and legend sheets. Subsequently, the extracted information is used to automatically generate the listing of all utilized instruments. The effectiveness of our method is evaluated on P&IDs from large industrial EPC projects, resulting in precision rates exceeding 98% and recall rates surpassing 99%. These results demonstrate the suitability of our method for industrial deployment. The successful application of our method has the potential to reduce engineering costs and increase the efficiency of EPC projects. Furthermore, the method could be adapted for additional applications in the EPC industry, which highlights the method's industrial value.

Platform ecosystems have revolutionized value creation across numerous industries, inducing technical leaps and scaling artifact generation in scales and speeds unattainable through traditional vertically integrated or single-firm models. These ecosystems rely on collaborative interactions among actors to co-create and reuse value. Such ecosystems are socio-technical environments which require complex governance and orchestration strategies to ensure ecosystem health and performance from financial, technical, and social perspectives. Consequently, monitoring the performance of platform ecosystems requires non-primitive metrics as factors contributing to ecosystem performance are multifaceted compared to conventional software settings. Effective orchestration of platform ecosystems requires relies on access to real-time quantitative performance indicators. The existing literature offers various quantitative health metrics and performance indicators for platform ecosystems, yet these are dispersed across multiple studies and often embedded in abstract models or found within generic analytics systems. This research reviews existing quantitative real-time health metrics and performance indicators of platform ecosystems. We identified 417 distinct metrics after eliminating duplicates and incomplete definitions, and refined 168 of these metrics to be calculable in real-time using a consistent framework for definition, nomenclature, and quantification. Furthermore, we compiled existing real-time ecosystem health monitoring methods into a reference architecture and tested its feasibility in two active platform ecosystems. The study yields four key contributions: a practical catalog of platform ecosystem health metrics; a reference architecture for creating real-time ecosystem health monitoring solutions, demonstrated through implementation in two operational platform ecosystems; industry-relevant insights for practitioners; and a discussion of potential future research directions.

Engineering, Procurement, and Construction (EPC) projects often encounter critical inefficiencies in Interdisciplinary Check (IDC) processes, where engineers manually validate design consistency across drawings and specifications, resulting in bottlenecks in large-scale projects. This paper introduces a proof-of-concept(PoC) human-AI cooperative framework for automating IDC workflows. The framework combines deep learning for document processing, heuristic algorithms for feature extraction, and a Large Multimodal Model (LMM) for cross-domain validation, while preserving engineering decision authority. The framework extracts key engineering data, links components across disciplines, and checks compliance with industry standards and client requirements. Testing on controlled data from a single EPC project (with 25 different engineering drawings) successfully identified all design discrepancies within the predefined validation scope, demonstrating notable workflow efficiency potential. As Phase 1 of a three-phase research cycle, this confirms technical feasibility before expanding to multiple projects and company-wide deployment, showing how human-AI collaboration can revolutionize engineering validation while preserving oversight of engineering expertise.

Interpreting complex engineering drawings requires substantial manual effort in industrial workflows, with engineers spending hundreds of hours verifying correlations between visual elements and structured specifications across thousands of documents. This challenge is particularly acute in the Engineering, Procurement and Construction (EPC) industries, where interpretation errors are propagated into costly procurement and construction mistakes. We propose a cybernetic systems approach using Large Multimodal Models (LMMs) as cognitive partners in engineering documentation workflows, enhancing human capabilities through intelligent assistance in verification and information extraction tasks. To validate this solution, we systematically evaluated five LMMs on 60 piping isometric drawings with varying template structures, measuring both optical character recognition accuracy and correlation capabilities between drawings and their associated Bills of Materials. The results demonstrated significant performance variation between systems, with Claude 3.5 Sonnet and GPT-4o achieving greater accuracy 70%, while open source alternatives faced challenges with complex layouts and ambiguous visual information. These findings establish benchmarks for humanmachine collaboration in engineering documentation processing and provide a framework for integrating intelligent systems into technical workflows across multiple industrial domains.

Context: Machine Learning Operations (MLOps) has become a top priority for companies. However, its adoption has become challenging due to the need for proper guidance and awareness. Most of the MLOps solutions available in the market are designed to fit the specific platform, tools and culture of the providers. Objective: The objective is to develop a structured approach to adopting, assessing and advancing MLOps adoption. Methods: The study was conducted based on a multi-case study across fourteen companies. Results: We provide a comprehensive analysis that highlights the similarities and differences in the adoption of MLOps practices among companies. We have also empirically validated the developed MLOps framework and MLOps maturity model. Furthermore, we carefully reviewed the feedback received from practitioners and revised the MLOps framework and maturity model to confirm its effectiveness. Additionally, we develop an MLOps taxonomy for classifying ML use cases based on their context and requirements into the desired stage of the MLOps framework and maturity model. Conclusion: The findings provide companies with a structured approach to adopt, assess, and further advance the adoption of MLOps practices regardless of their current status.

Context: The role of product management (PM) is key for building, implementing and managing software-intensive systems. Whereas engineering is concerned with how to build systems, PM is concerned with ‘what’ to build and ‘why’ we should build the product. The role of PM is recognized as critical for the success of any product. However, few studies explore how the role of PM is changing due to recent trends that come with digitalization and digital transformation. Objectives: Although there is prominent research on PM, few studies explore how this role is changing due to the digital transformation of the software-intensive industry. In this paper, we study how trends such as DevOps and short feedback loops, data and artificial intelligence (AI), as well as the emergence of digital ecosystems, are changing current product management practices. Methods: This study employs a qualitative approach using multi-case study research as the method. For our research, we selected five case companies in the software-intensive systems domain. Through workshop sessions, frequent meetings and interviews, we explore how DevOps and short feedback loops, data and artificial intelligence (AI), and digital ecosystems challenge current PM practices. Results: Our study yielded an in-depth understanding of how digital transformation of the software-intensive systems industry is changing current PM practices. We present empirical results from workshops and from interviews in which case company representatives share their insights on how software, data and AI impact current PM practices. Based on these results, we present a framework organized along two dimensions, i.e. a certainty dimension and an approach dimension. The framework helps structure the approaches product managers can employ to select and prioritize development of new functionality. Contributions: The contribution of this paper is a framework for ‘Strategic Digital Product Management’ (SDPM). The framework outlines nine approaches that product managers can employ to maximize the return on investment (RoI) of R&D using new digital technologies.

Federated Learning (FL) has gained prominence as a solution for preserving data privacy in machine learning applications. However, existing FL frameworks pose challenges for software engineers due to implementation complexity, limited customization options, and scalability issues. These limitations prevent the practical deployment of FL, especially in dynamic and resource-constrained edge environments, preventing its widespread adoption. To address these challenges, we propose EdgeFL, an efficient and low-effort FL framework designed to overcome centralized aggregation, implementation complexity and scalability limitations. EdgeFL applies a decentralized architecture that eliminates reliance on a central server by enabling direct model training and aggregation among edge nodes, which enhances fault tolerance and adaptability to diverse edge environments. We conducted experiments and a case study to demonstrate the effectiveness of EdgeFL. Our approach focuses on reducing weight update latency and facilitating faster model evolution on edge devices. Our findings indicate that EdgeFL outperforms existing FL frameworks in terms of learning efficiency and performance. By enabling quicker model evolution on edge devices, EdgeFL enhances overall efficiency and responsiveness to changing data patterns. EdgeFL offers a solution for software engineers and companies seeking the benefits of FL, while effectively overcoming the challenges and privacy concerns associated with traditional FL frameworks. Its decentralized approach, simplified implementation, combined with enhanced customization and fault tolerance, make it suitable for diverse applications and industries.

ABSTRACT Manual analysis of diagrams and legend sheets in engineering projects is time consuming and needs automation. The lack of standardized legend formats complicates creating a general method for automated information extraction. Existing approaches require training and custom rules for each project. This study proposes a novel solution combining optical character recognition with vision language models and multimodal prompt engineering to automate information extraction from diverse legend sheets without training. It integrates legend information with information extracted from diagrams, unlike studies that only focus on diagrams. Our study shows that VLMs, guided by multimodal prompts, can accurately extract information from diverse legend sheets, enabling automatic information extraction in diagrams across engineering projects. We validate our method through a case study involving the extraction of instruments from piping and instrumentation diagrams (P&IDs) and their legends across three projects with varied formats and standards. The proposed method achieved 100% accuracy in legend classification and information extraction, and 99.68% precision and 95.91% recall in generating instrument listings. The results demonstrate the effectiveness of our approach, significantly enhancing the accuracy and efficiency of information extraction from diagrams. This method can be adapted to different legend formats and diagrams, providing a versatile solution for various industries.

Continuous deployment aims to reduce the deployment cycle of new software, enabling software development companies to deliver new and improved functionalities to customers faster and more frequently. In software-intensive embedded systems, continuous deployment is often introduced as a subsequent step to continuous integration. While several empirical studies explored the transition from continuous integration to continuous deployment from several aspects, such as challenges, benefits, and success factors, these studies remain limited as they focus on applying continuous deployment to a small subset of the entire customer base. However, as software-intensive embedded systems are often high-volume products used and operated by many different customers, scaling continuous deployment becomes important to ensure that all customers perceive its benefits while at the same time enabling the software development organization to have one release and deployment cycle applicable to the entire customer base. Thus, to further understand the progression of continuous deployment from introduction to scaling, we conducted a longitudinal case study at a multinational telecommunications company producing complex telecommunications software-intensive embedded systems. Our results show that continuous deployment passes through four phases: R&D experiment, R&D core practice, as a service, and finally, continuous deployment supporting a result-oriented business model. Based on the results, we inductively derive a maturity model for continuous deployment, which we discuss based on the four dimensions of the BAPO framework (Business, Architecture, Process, and Organization).

Federated Learning (FL) has emerged as a promising approach for collaborative machine learning, addressing data privacy concerns. As data security and privacy concerns continue to gain prominence, FL stands out as an option to enable organizations to leverage collective knowledge without compromising sensitive data. However, existing FL platforms and frameworks often present challenges for software engineers in terms of complexity, limited customization options, and scalability limitations. In this paper, we introduce EdgeFL, an edge-only lightweight decentralized FL framework, designed to overcome the limitations of centralized aggregation and scalability in FL deployments. By adopting an edge-only model training and aggregation approach, EdgeFL eliminates the need for a central server, enabling seamless scalability across diverse use cases. Our results show that EdgeFL reduces weights update latency and enables faster model evolution, enhancing the efficiency of edge model learning. Moreover, EdgeFL exhibits improved classification accuracy compared to traditional centralized FL approaches. By leveraging EdgeFL, software engineers can harness the benefits of Federated Learning while overcoming the challenges associated with existing FL platforms/frameworks.

DevOps has become a widely adopted approach in the software industry, especially among companies developing web-based applications. The main focus of DevOps is to address social and technical bottlenecks along the software flow, from the developers' code changes to delivering these changes to the production environments used by customers. However, DevOps does not consider the software flow's content, e.g., new features, bug fixes, or security patches, and the customer value of each content. In addition, DevOps assumes that a streamlined software flow leads to a continuous value flow, as customers use the new software and extract value-adding content intuitively. However, in a Software-intensive System of Systems (SiSoS), customers need to understand the content of the software flow to validate, test, and adopt their operation procedures before using the new software. Thus, while DevOps has been extensively studied in the context of web-based applications, its adoption in SiSoS is a relatively unexplored area. Therefore, we conducted a case study at a multinational telecommunications provider focusing on 5G systems. Our findings reveal that DevOps has three sub-flows: legacy, feature, and solution. Each sub-flow has distinct content and customer value, requiring a unique approach to extracting it. Our findings highlight the importance of understanding the software flow's content and how each content's value can be extracted when adopting DevOps in SiSoS.

Context: Software ecosystems have transformed many industries, redefining collaboration and value co-creation. The success of such ecosystems depends on the dynamism of the network of users on its different sides. Consequently, decision-making in such multifaceted and interconnected environments is more complex than in conventional software products. On-line controlled experiments are considered the gold standard for aiding decision-making in software engineering processes. Experiments are extensively used to reduce bias and estimation noise for design, engineering, and business decisions. However, experimentation in software ecosystems is inherently more com-plex as it deals with atypical sources of bias and technical complications. Primary studies of experimentation approaches in software ecosystems are scattered across multiple domains and disciplines, and secondary research on the topic is scarce as highlighted in different tertiary studies. Hence, we conducted this study. Objectives: To explore primary research on experimentation in software ecosystems; Summarize current approaches, toolboxes, and solutions that practitioners and researchers, facing similar problems, can use to inform their approaches; To outline underexplored research areas and provide recommendations for practitioners. Method: We conducted a systematic literature review. The search strategy, application of exclusion and inclusion criteria, and subsequent quality assessment resulted in 63 relevant studies. Data extraction process was designed and carried out to collect data relevant to the study objectives. The extracted data under-went descriptive and thematic syntheses and analyses, in addition to cross-analysis on relevant axes. Contributions: The study resulted in four contributions. First, a distillation of the themes and patterns in the available research on the topic. Second, a practical summary of the experimental designs specific to each software ecosystem type. Third, an actionable road map for practitioners in order to achieve exper-imentation maturity in software ecosystems. Fourth, an outline of the underexplored research areas.

Industrial software ecosystems refer to a network of interdependent actors, co-creating value through a shared technological platform specifically tailored to industrial sectors. Developing, maintaining, and orchestrating such platforms involves many challenges that require complex decision making. Experimentation can help alleviate this complexity and reduce decision uncertainty and bias. However, experimentation requires certain organizational, infrastructural, and data-related prerequisites which can be uniquely challenging to achieve in industrial software ecosystems. Through semi-structured interviews with 25 industry professionals involved in various roles across 17 ecosystems, we analyze the difficulties faced in conducting effective experiments in such environments. The interview protocol covered aspects related to the methodologies, data handling processes, and current experimentation practices, as well as the challenges faced by practitioners who engage in experimentation initiatives. The study findings reveal technical, organizational, and market-related challenges, detailing the complexities facing experimentation initiatives in industrial software ecosystems. The findings are presented in an actionable manner, following a model that allows business-oriented alignment of architecture, process, and organizational evolution strategies. The study identifies key impediments, such as data integration difficulties, stringent regulatory environments, and prevailing organizational cultures that hinder continuous experimentation practices. Our analysis provides a foundation for understanding the unique challenges facing experimentation efforts in industrial software ecosystems and offers insights into potential strategies to improve the effectiveness of these initiatives.

Data is key for rapid and continuous delivery of customer value. By collecting data from products in the field, companies in the embedded systems domain can measure and monitor product performance and they get the opportunity to provide customers with insights and data-driven services. However, while the notion of data-driven development is not new, embedded systems companies are facing a situation in which data volumes are growing exponentially and this is not without its challenges. Suddenly, the cost of collecting, storing and processing data becomes a concern and while there is prominent research on different aspects of data-driven development, there is little guidance for how to reason about business value versus costs of data. In this paper, we present findings from case study research conducted in close collaboration with four companies in the embedded systems domain. The contribution of this paper is a framework that provides a holistic understanding of the multiple dimensions that need to be considered when reasoning about business value versus cost of collecting, storing and processing data.

The success of any Engineering, Procurement, and Construction (EPC) project depends on the engineering deliverables developed during project execution. An important deliverable is the Line List document, produced by extracting pipeline numbers from Piping and Instrumentation Diagrams (P&IDs). As the creation of this document is time-consuming, the automation of this process could reduce manual engineering work. However, the complexity of the P&IDs renders traditional computer vision approaches unsuitable. Therefore, deep learning text detection could be utilized to achieve this task. This study assessed the applicability of text detection methods for automating pipeline number information extraction in P&IDs. Our findings indicate that the methods previously used to detect text on P&IDs have limitations in accurately capturing the entire line numbers. Furthermore, we propose a line number detection method achieving a recall rate of over 90% on our evaluation data, consisting of P&IDs from diverse industrial projects. Thus, we demonstrate our method's generalizability to different line number formats and its potential for industrial application. Moreover, the proposed method can be adapted to other types of engineering drawings beyond P&IDs. Thus, it could be used in additional applications for digitizing engineering drawings.

Applied Artificial Intelligence (AI) in engineering is gaining significant traction. AI object detection methods can be applied in the engineering industry to extract information from engineering drawings, offering immense benefits to engineers. A promising application of AI in industrial engineering is symbol recognition applied to engineering drawings. However, these drawings often exhibit areas with a high density of symbols, as well as noise in the form of markups, indicating revisions. These factors could cause symbol misclassification or omission, impacting applications reliant on accurate symbol recognition. This study evaluates the accuracy of a symbol recognition model on engineering drawings called Piping and Instrumen-tation Diagrams (P &IDs) exhibiting varying levels of density and markups causing noise. Despite the assumption that density poses a challenge for accurate symbol recognition in engineering drawings, our study reveals that density has no significant impact on recognition performance when a dense detector is employed. In addition, we quantitatively show that markup-induced noise on engineering drawings negatively influences recognition accuracy. Finally, we provide recommendations regarding the applicability of symbol recognition in engineering applications. The study's findings and recommendations apply to any P &IDs, regardless of the standard used, as they were evaluated on various worldwide projects. Moreover, the research not only contributes to the advancement of symbol recognition on P&IDs, but also can be applied to other types of engineering drawings. Thus, it holds the potential for enhancing symbol recognition in various real-world industrial applications and research.

In the Engineering, Procurement, and Construction (EPC) sector, accurate cost estimations during the tendering phase are crucial for maintaining competitiveness, especially with constrained project schedules and rising labor expenses. Typically, these estimations are labor-intensive, relying heavily on manual evaluations of engineering drawings, which are often shared in PDF format due to intellectual property concerns. This study introduces an innovative solution tailored for the energy industry, utilizing Artificial Intelligence (AI) - primarily deep learning (DL) and machine learning (ML) techniques - to streamline material quantity estimation, thereby saving engineering time and costs. Built on empirical data from a large EPC company operating in the energy sector, AI-based product development experiences, and academic research, our approach aims to enhance the efficiency and accuracy of engineering work, promoting better decision-making and resource distribution. While our focus is on enhancing a particular activity within the case company using AI, the method's broader applicability in the EPC sector potentially benefits both industry professionals and researchers. This study not only advances a practical application but also provides valuable insights for those seeking to develop AI -driven solutions across various engineering disciplines.

The Engineering, Procurement and Construction (EPC) businesses operating within the energy sector are recognizing the increasing importance of Artificial Intelligence (AI). Many EPC companies and their clients have realized the benefits of applying AI to their businesses in order to reduce manual work, drive productivity, and streamline future operations of engineered installations in a highly competitive industry. The current AI market offers various solutions and services to support this industry, but organizations must understand how to acquire AI technology in the most beneficial way based on their business strategy and available resources. This paper presents a framework for EPC companies in their transformation towards AI. Our work is based on examples of project execution of AI-based products development at one of the biggest EPC contractors worldwide and on insights from EPC vendor companies already integrating AI into their engineering solutions. The paper covers the entire life cycle of building AI solutions, from initial business understanding to deployment and further evolution. The framework identifies how various factors influence the choice of approach toward AI project development within large international engineering corporations. By presenting a practical guide for optimal approach selection, this paper contributes to the research in AI project management and organizational strategies for integrating AI technology into businesses. The framework might also help engineering companies choose the optimum AI approach to create business value.

Abstract Continuous deployment has been practiced for many years by companies developing web‐ and cloud‐based applications. To succeed with continuous deployment, these companies have a strong collaboration culture between the operations and development teams. In addition, these companies use AI, analytics, and big data to assist with time‐consuming postdeployment activities such as continuous monitoring and fault identification. Thus, the term AIOps has evolved to highlight the importance and difficulty of maintaining highly available applications in a complex and dynamic environment. In contrast, software‐intensive embedded systems often provide customer product‐related services, such as maintenance, optimization, and support. These services are critical for these companies as they provide significant revenue and increase customer satisfaction. Therefore, the objective of our study is to gain an in‐depth understanding of the impact of continuous deployment on product‐related services provided by software‐intensive embedded systems companies. In addition, we aim to understand how AIOps can support continuous deployment in the context of software‐intensive embedded systems. To address this objective, we conducted a case study at a large and multinational telecommunications systems provider focusing on the radio access network (RAN) systems for 4G and 5G networks. The company provides RAN products and three complementing services: rollout, optimization, and customer support. The results from the case study show that the boundaries between product‐related services become blurry with continuous deployment. In addition, product‐related services, which were conducted in sequence by independent projects, converge with continuous deployment and become part of the same project. Further, AIOps platforms play an important role in reducing costs and increasing postdeployment activities' efficiency and speed. These results show that continuous deployment has a profound impact on the software‐intensive system's provider service organization. The service organization becomes the connection between the R&D organization and the customer. In order to cope with the increased speed of releases, deployment and postdeployment activities need to be largely automated. AIOps platforms are seen as a critical enabler in managing the increasing complexity without increasing human involvement.

Machine Learning Operations (MLOps) play a crucial role in the success of data science projects in companies. However, despite its obvious benefits, several companies struggle to adopt MLOps practices and face difficulty in deciding how to deploy and evolve ML models. To gain a deeper understanding of these challenges, we conduct a multi-case study involving nine practitioners from seven companies. Based on our empirical results, we identify the key trade-offs we see companies make when adopting MLOps. We categorise these trade-offs into four concerns of the BAPO model: Business, Architecture, Process, and Organisation. Finally, we provide suggestions to mitigate the identified trade-offs. By identifying and detailing these trade-offs and the implications of these, this research helps companies to ensure the successful adoption of MLOps.

Data pipelines can be defined as a complex chain of interconnected activities that starts with a data source and ends in a data sink. They can process data in multiple formats from various data sources with minimal human intervention, speed up data life cycle operations, and enhance productivity in data-driven organizations. As a result, companies place a high value on strengthening the maturity of their data pipelines. The available literature, on the other hand, is significantly insufficient in terms of providing a comprehensive roadmap to guide companies in assessing the maturity of their data pipelines. Therefore, this case study focuses on developing a data pipeline maturity assessment model that can evaluate the maturity of data pipelines in a staged manner from maturity level 1 to maturity level 5. We conducted empirical research in order to develop the maturity assessment model on the basis of five different determinants to address the specific needs of each data pipeline maturity level. Accordingly, it aims to support organizations in assessing their current data pipeline maturity, determining challenges at each stage, and preparing an extensive roadmap and suggestions for data pipeline maturity improvement. In future work, we plan to employ the maturity model in different companies as a case study to evaluate its applicability and usefulness.

Deep reinforcement learning has advanced signifi-cantly in recent years, and it is now used in embedded systems in addition to simulators and games. Reinforcement Learning (RL) algorithms are currently being used to enhance device operation so that they can learn on their own and offer clients better services. It has recently been studied in a variety of industrial applications. However, reinforcement learning, especially when controlling a large number of agents in an industrial environment, has been demonstrated to be unstable and unable to adapt to realistic situations when used in a real-world setting. To address this problem, the goal of this study is to enable multiple reinforcement learning agents to independently learn control policies on their own in dynamic industrial contexts. In order to solve the problem, we propose a dynamic multi-agent reinforcement learning (dynamic multi-RL) method along with adaptive exploration (AE) and vector-based action selection (VAS) techniques for accelerating model convergence and adapting to a complex industrial environment. The proposed algorithm is tested for validation in emergency situations within the telecommunications industry. In such circumstances, three unmanned aerial vehicles (UAV-BSs) are used to provide temporary coverage to mission-critical (MC) customers in disaster zones when the original serving base station (BS) is destroyed by natural disasters. The algorithm directs the participating agents automatically to enhance service quality. Our findings demonstrate that the proposed dynamic multi-RL algorithm can proficiently manage the learning of multiple agents and adjust to dynamic industrial environments. Additionally, it enhances learning speed and improves the quality of service.

Companies producing software-intensive products do not only offer products to customers but Product Service Systems (PSS), a combination of the products and services that address customers’ needs. Further, product-related services are key in ensuring customer satisfaction as the service organization represents the company’s interface toward its customers, who operate and use the products. Therefore, while DevOps has been widely adopted in companies developing web-based applications aiming to streamline the Development and Operations activities, the projecting of DevOps as applied in web-based applications to PSS is difficult without considering the role of services. Therefore, based on a two years participant observation case study conducted at a multinational telecommunications systems provider, we propose a new and novel approach called Development-Services-Operations (DevServOps) which incorporates services as a key player facilitating an end-to-end software flow toward customers in one direction and feedback toward developers in the other direction.

In recent years, the interest in applying machine learning (ML) and deep learning (DL) has been increasing due to their ability to learn to predict and find structure in data. The most common approach of ML and DL is supervised learning. Supervised learning requires the input data to be labeled. However, as reported by many industries, such as the embedded systems domain, fully labeled datasets are difficult to obtain since data labeling is manually intensive. This paper uses a semi-supervised learning approach on real-world Pulse-Doppler data obtained from our industry collaborator Saab to address this challenge. We took inspiration from the FixMatch algorithm. To investigate whether unlabeled data can help improve classification accuracy, we compare FixMatch to a supervised baseline. We use five different settings for the number of available labels per class label to investigate how many labeled instances and how much manual effort is required for optimal accuracy. Bayesian Linear Regression is used to analyze the results. The results show that FixMatch can reach a higher accuracy than the supervised baseline. Furthermore, FixMatch requires more computation time but will help reduce manual effort. In addition, FixMatch will not underfit or overfit. Thanks to this study, practitioners know the benefits of utilizing FixMatch and when it is safe to use to improve a supervised baseline in the industry.

The emergence of platform ecosystems has transformed the business landscape in many industries, giving rise to novel modes of interorganizational cooperation and value co-creation, as well as unconventional challenges. The vast traces of data generated by platform ecosystems makes them ripe for the use of analytics and data-driven methods aimed at improving their health, performance, business outcomes, and evolution. However, the research on the application of analytics within platform ecosystems is limited and spread across multiple disciplines. To address this gap, we conducted a systematic literature review on the application of analytics and data-driven methods and practices within platform ecosystems. A total of 56 studies were reviewed, and underwent data extraction, analysis, and synthesis processes. In addition to presenting themes and patterns in the recent and relevant literature on platform ecosystems analytics, our review offers the following outcomes: an actionable overview of the analytics toolbox currently used within platform ecosystems—spanning domains such as machine learning, deep learning, data science, modelling, simulation, among others—; a roadmap for practitioners to achieve analytics maturity; and a summary of underexplored research areas.

As industries continue to digitize at a rapid pace, the number of integrations, the connections between different digital systems, is increasing. As a consequence, integration management is becoming increasingly difficult, leading to scalability, stability, and system outage problems that directly affect company performance. For that reason the modern integration management platforms have emerged to offer efficient management of the growing number of integrations, as well as a new way to handle, enrich, and utilize data both inside and outside organizations.This research involved interviews with 20 software vendor professionals and 20 clients who had recently undergone an integration platform project. The study identified key drivers behind integration management projects and recognized different maturity levels in integration management. The results revealed that more mature companies are utilizing modern integration platforms to support their digital transformation. The study also highlighted the growing importance of data management and utilization in integration solutions.To address these findings, a maturity model was developed to help companies evaluate their current integration management status and find ways to progress in their integration management.

Companies across various domains increasingly adopt Machine Learning Operations (MLOps) as they recognise the significance of operationalising ML models. Despite growing interest from practitioners and ongoing research, MLOps adoption in practice is still in its initial stages. To explore the adoption of MLOps, we employ a multi-case study in seven companies. Based on empirical findings, we propose a maturity model outlining the typical stages companies undergo when adopting MLOps, ranging from Ad hoc to Kaizen. We identify five dimensions associated with each stage of the maturity model as part of our MLOps framework. We also map these seven companies to the identified stages in the maturity model. Our study serves as a roadmap for companies to assess their current state of MLOps, identify gaps and overcome obstacles to successfully adopting MLOps.

Effective collection and use of data is key for companies across domains and it is only increasing in importance. For companies in the embedded systems domain, data constitutes the basis not only for quality assurance and diagnostics of their systems but also for new service development and innovation. For these companies, data is an enabler for continuous delivery of customer value and hence, a key asset for entirely new and recurring revenue streams. However, effective use of data requires careful collection of different kinds of data depending on the purpose and context for which it is intended to be used. In this paper, we identify the challenges that companies experience in their contemporary data practices and we outline the kinds of data that companies need to collect as they evolve through different maturity stages. In addition, we provide concrete guidance on the specific data to collect during each maturity stage.

In recent years, Federated Learning, as an approach to distributed learning, has shown its potential with the increasing number of devices on the edge and the development of computing power. The method enables large-scale training on the device that creates the data but with the sensitive data remaining within the data’s owner. In reality, however, the vast majority of enterprises have the problem of low data volume and poor model quality to support the implementation of Federated Learning methods. Learning quality assurance for edge devices is still the major issue which prevents Federated Learning to be applied in industrial contexts, especially in safety-critical applications. In this paper, we propose a quality-based asynchronous Federated Learning algorithm (QuaFedAsync) to address these challenges. We report on a study in which we used two well-known data sets, i.e., DDAD and KITTI datasets, and validate the proposed algorithm on an industrial use case concerned with monocular depth estimation in the automotive domain. Our results show that the proposed algorithm significantly improves the prediction performance compared to the commonly applied aggregation protocols while maintaining the same level of accuracy as centralized machine learning. Based on the results, we prove the learning efficiency and robustness when applying the algorithm to industrial scenarios.

Abstract Artificial intelligence is increasingly becoming important to businesses because many companies have realized the benefits of applying machine learning (ML) and deep learning (DL) in their operations. ML and DL have become attractive technologies for organizations looking to automate repetitive tasks to reduce manual work and free up resources for innovation. Unlike rule‐based automation, typically used for standardized and predictable processes, machine learning, especially deep learning, can handle more complex tasks and learn over time, leading to greater accuracy and efficiency improvements. One of such promising applications is to use AI to reduce manual engineering work. This paper discusses a particular case within McDermott where the research team developed a DL model to do a quality check of complex blueprints. We describe the development and the final product of this case—AI‐based software for the engineering, procurement, and construction (EPC) industry that helps to find the design mistakes buried inside very complex engineering drawings called piping and instrumentation diagrams (P&IDs). We also present a cost‐benefit analysis and potential scale‐up of the developed software. Our goal is to share the successful experience of AI‐based product development that can substantially reduce the engineering hours and, therefore, reduce the project's overall costs. The developed solution can also be potentially applied to other EPC companies doing a similar design for complex installations with high safety standards like oil and gas or petrochemical plants because the design errors it captures are common within this industry. It also could motivate practitioners and researchers to create similar products for the various fields within engineering industry.

Abstract Artificial intelligence (AI) and the use of machine learning (ML) and deep learning (DL) technologies are becoming increasingly popular in companies. These technologies enable companies to leverage big quantities of data to improve system performance and accelerate business development. However, despite the appeal of ML/DL, there is a lack of systematic and structured methods and processes to help data scientists and other company roles and functions to develop, deploy and evolve models. In this paper, based on multi‐case study research in six companies, we explore practices and challenges practitioners experience in developing ML/DL models as part of large software‐intensive embedded systems. Based on our empirical findings, we derive a conceptual framework in which we identify three high‐level activities that companies perform in parallel with the development, deployment and evolution of models. Within this framework, we outline activities, iterations and triggers that optimize model design as well as roles and company functions. In this way, we provide practitioners with a blueprint for effectively integrating ML/DL model development into the business to achieve better results than other (algorithmic) approaches. In addition, we show how this framework helps companies solve the challenges we have identified and discuss checkpoints for terminating the business case.

Abstract Continuous experimentation (CE) refers to a set of practices used by software companies to rapidly assess the usage, value, and performance of deployed software using data collected from customers and systems in the field using an experimental methodology. However, despite its increasing popularity in developing web‐facing applications, CE has not been studied in the development process of business‐to‐business (B2B) mission‐critical systems. By observing the CE practices of different teams, with a case study methodology inside Ericsson, we were able to identify the different practices and techniques used in B2B mission‐critical systems and a description and classification of the four possible types of experiments. We present and analyze each of the four types of experiments with examples in the context of the mission‐critical long‐term evolution (4G) product. These examples show the general experimentation process followed by the teams and the use of the different CE practices and techniques. Based on these examples and the empirical data, we derived the HURRIER process to deliver high‐quality solutions that the customers value. Finally, we discuss the challenges, opportunities, and lessons learned from applying CE and the HURRIER process in B2B mission‐critical systems.

Artificial intelligence is increasingly becoming important to businesses since many companies have realized the benefits of applying Machine Learning (ML) and Deep Learning (DL) into their operations. Nevertheless, ML/DL technologies' industrial development and deployment examples are still rare and generally confined within a small cluster of large international companies who are struggling to apply ML more broadly and deploy their use cases at a large scale. Meanwhile, current AI market has started offering various solutions and services. Thus, organizations must understand how to acquire AI technology based on their business strategy and available resources. This paper discusses the industrial experience of developing and deploying ML/DL use cases to support organizations in their transformation towards AI. We identify how various factors, like cost, schedule, and intellectual property, can be affected by the choice of approach towards ML/DL project development and deployment within large international engineering corporations. As a research result, we present a framework that covers the trade-offs between those various factors and can support engineering companies to choose the best approach based on their long-term business strategies and, therefore, would help to accomplish their ML/DL project deployment successfully.

Supporting customers after they acquire the prod-uct is essential for companies producing and selling software-intensive embedded systems products. Generally, customer sup-port is the first interaction point between the product users and the product vendor. Customer support is often engaged with answering customers' questions, troubleshooting, fault identification, and fixing product faults. While continuous deployment advocates for closer cooperation between the ones operating the software and the ones developing it, the means of such collaboration in general and the role of customer support, in particular, has not been addressed in the context of software-intensive embedded systems. Therefore, to better understand the impact that continuous deployment has on customer support and the role customer support should play in this context, we conducted a case study at a multinational company developing and selling telecommunications networks infrastructure. We focused on the 4th and 5th Generation (4G and 5G) Radio Access Networks (RAN) products, which can be considered a high volume product as they cover more than 80% of the world's population. Our study reveals that customer support needs to transition from a transaction-based and passive function triggered by customer support requests, to take an active role characterized by being proactive and preemptive to cope with the shorter operational time of a software version introduced by continuous deployment. In addition, customer support plays an essential role in making the feedback actionable by aggregating and consolidating feedback data to the R&D organization.

AI in the Engineering, Procurement and Construction (EPC) industry has not yet a proven track record in large-scale projects. Since AI solutions for industrial applications became available only recently, deployment experience and lessons learned are still to be built up. Several research papers exist describing the potential of AI, and many surveys and white papers have been published indicating the challenges of AI deployment in the EPC industry. However, there is a recognizable shortage of in-depth studies of deployment experience in academic literature, particularly those focusing on the experiences of EPC companies involved in large-scale project execution with high safety standards, such as the petrochemical or energy sector. The novelty of this research is that we explore in detail the challenges and obstacles faced in developing and deploying AI in a large-scale project in the EPC industry based on real-life use cases performed in an EPC company. Those identified challenges are not linked to specific technology or a company's know-how and, therefore, are universal. The findings in this paper aim to provide feedback to academia to reduce the gap between research and practice experience. They also help reveal the hidden stones when implementing AI solutions in the industry.

Modern release engineering practices such as continuous integration and delivery have allowed software development companies to transition from a long release cycle to a shorter one. The shorter release cycle has led to more software releases available to customers. At the same time, companies developing high-volume software-intensive embedded systems often deliver patch releases and maintenance releases on top of major and minor releases to customers who pick and choose what releases apply to them and decide when to upgrade the system, if to upgrade at all. While release engineering has been studied before in web-based, desktop-based, and embedded software, the focus has been on pre-release activities. Few studies have investigated what happens after the release, particularly the role of tracing software from release to deployment in high-volume software-intensive embedded systems. To address this gap, we conducted a qualitative case study at a multi-national telecommunications systems provider focusing on Radio Access Network (RAN) software. RAN software is a complex and large-scale embedded software used in mobile networks Base Stations (BS), providing software functionality for RAN mobile technologies ranging from 2G to 5G. Our study shed light on post-release software traceability and how it is used in the release engineering process.

For companies in the embedded systems domain, digitalization and digital technologies allow endless opportunities for new business models and continuous value delivery. While physical products still provide the core revenue, these are rapidly being complemented with offerings that allow for recurring revenue and that are based on software, data and artificial intelligence (AI). However, while new digital offerings allow for fundamentally new and recurring revenue streams and continuous value delivery to customers, the creation of these proves to be a challenging endeavour. In this paper, we study how companies explore ways to create new or additional value with the intention to complement their product portfolio with offerings that allow for recurring revenue. Based on multi-case study research, we identify the key challenges that companies in the embedded systems domain experience and we derive four organizational patterns that we see slow down innovation. Second, we present a framework outlining alternative types of offerings to customers. Third, we provide a value taxonomy in which we detail the different types of offerings and the value these provide to customers. For each value offering, we indicate whether this offering is (1) static or evolving, (2) bundled or unbundled, (3) free or monetized, and we provide examples from the case companies we studied.

Reinforcement learning, particularly deep reinforcement learning, has made remarkable progress in recent years and is now used not only in simulators and games but is also making its way into embedded systems as another software-intensive domain. However, when implemented in a real-world context, reinforcement learning is typically shown to be fragile and incapable of adapting to dynamic environments. In this paper, we provide a novel dynamic reinforcement learning algorithm for adapting to complex industrial situations. We apply and validate our approach using a telecommunications use case. The proposed algorithm can dynamically adjust the position and antenna tilt of a drone-based base station to maintain reliable wireless connectivity for mission-critical users. When compared to traditional reinforcement learning approaches, the dynamic reinforcement learning algorithm improves the overall service performance of a drone-based base station by roughly 20%. Our results demonstrate that the algorithm can quickly evolve and continuously adapt to the complex dynamic industrial environment.

Deep learning (DL) based software systems are difficult to develop and maintain in industrial settings due to several challenges. Data management is one of the most prominent challenges which complicates DL in industrial deployments. DL models are data-hungry and require high-quality data. Therefore, the volume, variety, velocity, and quality of data cannot be compromised. This study aims to explore the data management challenges encountered by practitioners developing systems with DL components, identify the potential solutions from the literature and validate the solutions through a multiple case study. We identified 20 data management challenges experienced by DL practitioners through a multiple interpretive case study. Further, we identified 48 articles through a systematic literature review that discuss the solutions for the data management challenges. With the second round of multiple case study, we show that many of these solutions have limitations and are not used in practice due to a combination of four factors: high cost, lack of skill-set and infrastructure, inability to solve the problem completely, and incompatibility with certain DL use cases. Thus, data management for data-intensive DL models in production is complicated. Although the DL technology has achieved very promising results, there is still a significant need for further research in the field of data management to build high-quality datasets and streams that can be used for building production-ready DL systems. Furthermore, we have classified the data management challenges into four categories based on the availability of the solutions.

Data collected from in-service products play an important role in enabling software-intensive embedded systems suppliers to embrace data-driven practices. Data can be used in many different ways such as to continuously learn and improve the product, enhance post-deployment services, reduce operational cost or create a better user experience. While there is no shortage of possible use cases leveraging data from in-service products, software-intensive embedded systems companies struggle to continuously collect data from their in-service products. Often, data collection is done in an ad-hoc way and targeting specific use cases or needs. Besides, few studies have investigated data collection challenges in relation to the data dimensions, which are the minimum set of quantifiable data aspects that can define software-intensive embedded product data from a collection point of view. To help address data collection challenges, and to provide companies with guidance on how to improve this process, we conducted a case study at a large multinational telecommunications supplier focusing on data characteristics and collection challenges from the Radio Access Networks (RAN) products. We further investigated the relations of these challenges to the data dimensions to increase our understanding of how data dominions contribute to the challenges.

Randomised field experiments, such as A/B testing, have long been the gold standard for evaluating the value that new software brings to customers. However, running randomised field experiments is not always desired, possible or even ethical in the development of automotive embedded software. In the face of such restrictions, we propose the use of the Bayesian propensity score matching technique for causal inference of observational studies in the automotive domain. In this paper, we present a method based on the Bayesian propensity score matching framework, applied in the unique setting of automotive software engineering. This method is used to generate balanced control and treatment groups from an observational online evaluation and estimate causal treatment effects from the software changes, even with limited samples in the treatment group. We exemplify the method with a proof-of-concept in the automotive domain. In the example, we have a larger control (Nc = 1100) fleet of cars using the current software and a small treatment fleet (Nt = 38), in which we introduce a new software variant. We demonstrate a scenario that shipping of a new software to all users is restricted, as a result, a fully randomised experiment could not be conducted. Therefore, we utilised the Bayesian propensity score matching method with 14 observed covariates as inputs. The results show more balanced groups, suitable for estimating causal treatment effects from the collected observational data. We describe the method in detail and share our configuration. Furthermore, we discuss how can such a method be used for online evaluation of new software utilising small groups of samples.

The impact of the Artificial Intelligence revolution is undoubtedly substantial in our society, life, firms, and employment. With data being a critical element, organizations are working towards obtaining high-quality data to train their AI models. Although data, data management, and data pipelines are part of industrial practice even before the introduction of ML models, the significance of data increased further with the advent of ML models, which force data pipeline developers to go beyond the traditional focus on data quality. The objective of this study is to analyze the impact of ML use cases on data pipelines. We assume that the data pipelines that serve ML models are given more importance compared to the conventional data pipelines. We report on a study that we conducted by observing software teams at three companies as they develop both conventional(Non-ML) data pipelines and data pipelines that serve ML-based applications. We study six data pipelines from three companies and categorize them based on their criticality and purpose. Further, we identify the determinants that can be used to compare the development and maintenance of these data pipelines. Finally, we map these factors in a two-dimensional space to illustrate their importance on a scale of low, moderate, and high.

The lack of labeled data is a major problem in both research and industrial settings since obtaining labels is often an expensive and time-consuming activity. In the past years, several machine learning algorithms were developed to assist and perform automated labeling in partially labeled datasets. While many of these algorithms are available in open-source packages, there is a lack of research that investigates how these algorithms compare to each other for different types of datasets and with different percentages of available labels. To address this problem, this paper empirically evaluates and compares seven algorithms for automated labeling in terms of their accuracy. We investigate how these algorithms perform in twelve different and well-known datasets with three different types of data, images, texts, and numerical values. We evaluate these algorithms under two different experimental conditions, with 10% and 50% labels of available labels in the dataset. Each algorithm, in each dataset for each experimental condition, is evaluated independently ten times with different random seeds. The results are analyzed and the algorithms are compared utilizing a Bayesian Bradley-Terry model. The results indicate that the active learning algorithms using the query strategies uncertainty sampling, QBC and random sampling are always the best algorithms. However, this comes with the expense of increased manual labeling effort. These results help machine learning practitioners in choosing optimal machine learning algorithms to label their data.

A/B experimentation is a known technique for data-driven product development and has demonstrated its value in web-facing businesses. With the digitalisation of the automotive industry, the focus in the industry is shifting towards software. For automotive embedded software to continuously improve, A/B experimentation is considered an important technique. However, the adoption of such a technique is not without challenge. In this paper, we present an architecture to enable A/B testing in automotive embedded software. The design addresses challenges that are unique to the automotive industry in a systematic fashion. Going from hypothesis to practice, our architecture was also applied in practice for running online experiments on a considerable scale. Furthermore, a case study approach was used to compare our proposal with state-of-practice in the automotive industry. We found our architecture design to be relevant and applicable in the efforts of adopting continuous A/B experiments in automotive embedded software.

With the development and the increasing interests in ML/DL fields, companies are eager to apply Machine Learning/Deep Learning approaches to increase service quality and customer experience. Federated Learning was implemented as an effective model training method for distributing and accelerating time-consuming model training while protecting user data privacy. However, common Federated Learning approaches, on the other hand, use a synchronous protocol to conduct model aggregation, which is inflexible and unable to adapt to rapidly changing environments and heterogeneous hardware settings in real-world scenarios. In this paper, we present an approach to real-time end-to-end Federated Learning combined with a novel asynchronous model aggregation protocol. Our method is validated in an industrial use case in the automotive domain, focusing on steering wheel angle prediction for autonomous driving. Our findings show that asynchronous Federated Learning can significantly improve the prediction performance of local edge models while maintaining the same level of accuracy as centralized machine learning. Furthermore, by using a sliding training window, the approach can minimize communication overhead, accelerate model training speed and consume real-time streaming data, proving high efficiency when deploying ML/DL components to heterogeneous real-world embedded systems.

In practice, supervised learning algorithms require fully labeled datasets to achieve the high accuracy demanded by current modern applications. However, in industrial settings supervised learning algorithms can perform poorly because of few labeled instances. Semi-supervised learning (SSL) is an automatic labeling approach that utilizes complete labels to infer missing labels in partially complete datasets. The high number of available SSL algorithms and the lack of systematic comparison between them leaves practitioners without guidelines to select the appropriate one for their application. Moreover, each SSL algorithm is often validated and evaluated in a small number of common datasets. However, there is no research that examines what datasets are suitable for comparing different SSL algorihtms. The purpose of this paper is to empirically evaluate the suitability of the datasets commonly used to evaluate and compare different SSL algorithms. We performed a simulation study using twelve datasets of three different datatypes (numerical, text, image) on thirteen different SSL algorithms. The contributions of this paper are two-fold. First, we propose the use of Bayesian congeneric item response theory model to assess the suitability of commonly used datasets. Second, we compare the different SSL algorithms using these datasets. The results show that with except of three datasets, the others have very low discrimination factors and are easily solved by the current algorithms. Additionally, the SSL algorithms have overlapping 90% credible intervals, indicating uncertainty in the difference between the accuracy of these SSL models. The paper concludes suggesting that researchers and practitioners should better consider the choice of datasets used for comparing SSL algorithms.

The adoption of continuous software engineering practices such as DevOps (Development and Operations) in business operations has contributed to significantly shorter software development and deployment cycles. Recently, the term MLOps (Machine Learning Operations) has gained increasing interest as a practice that brings together data scientists and operations teams. However, the adoption of MLOps in practice is still in its infancy and there are few common guidelines on how to effectively integrate it into existing software development practices. In this paper, we conduct a systematic literature review and a grey literature review to derive a framework that identifies the activities involved in the adoption of MLOps and the stages in which companies evolve as they become more mature and advanced. We validate this framework in three case companies and show how they have managed to adopt and integrate MLOps in their large-scale software development companies. The contribution of this paper is threefold. First, we review contemporary literature to provide an overview of the state-of-the-art in MLOps. Based on this review, we derive an MLOps framework that details the activities involved in the continuous development of machine learning models. Second, we present a maturity model in which we outline the different stages that companies go through in evolving their MLOps practices. Third, we validate our framework in three embedded systems case companies and map the companies to the stages in the maturity model.

A/B testing is gaining attention in the automotive sector as a promising tool to measure causal effects from software changes. Different from the web-facing businesses, where A/B testing has been well-established, the automotive domain often suffers from limited eligible users to participate in online experiments. To address this shortcoming, we present a method for designing balanced control and treatment groups so that sound conclusions can be drawn from experiments with considerably small sample sizes. While the Balance Match Weighted method has been used in other domains such as medicine, this is the first paper to apply and evaluate it in the context of software development. Furthermore, we describe the Balance Match Weighted method in detail and we conduct a case study together with an automotive manufacturer to apply the group design method in a fleet of vehicles. Finally, we present our case study in the automotive software engineering domain, as well as a discussion on the benefits and limitations of the A/B group design method.

In recent years, with more edge devices being put into use, the amount of data that is created, transmitted and stored is increasing exponentially. Moreover, due to the development of machine learning algorithms, modern software-intensive systems are able to take advantage of the data to further improve their service quality. However, it is expensive and inefficient to transmit large amounts of data to a central location for the purpose of training and deploying machine learning models. Data transfer from edge devices across the globe to central locations may also raise privacy and concerns related to local data regulations. As a distributed learning approach, Federated Learning has been introduced to tackle those challenges. Since Federated Learning simply exchanges locally trained machine learning models rather than the entire data set throughout the training process, the method not only protects user data privacy but also improves model training efficiency. In this paper, we have investigated an advanced machine learning algorithm, Deep Neural Decision Forests (DNDF), which unites classification trees with the representation learning functionality from deep convolutional neural networks. In this paper, we propose a novel algorithm, AF-DNDF which extends DNDF with an asynchronous federated aggregation protocol. Based on the local quality of each classification tree, our architecture can select and combine the optimal groups of decision trees from multiple local devices. The introduction of the asynchronous protocol enables the algorithm to be deployed in the industrial context with heterogeneous hardware settings. Our AF-DNDF architecture is validated in an automotive industrial use case focusing on road objects recognition and demonstrated by an empirical experiment with two different data sets. The experimental results show that our AF-DNDF algorithm significantly reduces the communication overhead and accelerates model training speed without sacrificing model classification performance. The algorithm can reach the same classification accuracy as the commonly used centralized machine learning methods but also greatly improve local edge model quality.

In recent years, with the development of computation capability in devices, companies are eager to investigate and utilize suitable ML/DL methods to improve their service quality. However, with the traditional learning strategy, companies need to first build up a powerful data center to collect and analyze data from the edge and then perform centralized model training, which turns out to be inefficient. Federated Learning has been introduced to solve this challenge. Because of its characteristics such as model-only exchange and parallel training, the technique can not only preserve user data privacy but also accelerate model training speed. The method can easily handle real-time data generated from the edge without taking up a lot of valuable network transmission resources. In this paper, we introduce an approach to end-to-end on-device Machine Learning by utilizing Federated Learning. We validate our approach with an important industrial use case in the field of autonomous driving vehicles, the wheel steering angle prediction. Our results show that Federated Learning can significantly improve the quality of local edge models and also reach the same accuracy level as compared to the traditional centralized Machine Learning approach without its negative effects. Furthermore, Federated Learning can accelerate model training speed and reduce the communication overhead, which proves that this approach has great strength when deploying ML/DL components to various real-world embedded systems.

Abstract With digitalization and with technologies such as software, data, and artificial intelligence, companies in the embedded systems domain are experiencing a rapid transformation of their conventional businesses. While the physical products and associated product sales provide the core revenue, these are increasingly being complemented with service offerings, new data‐driven services, and digital products that allow for continuous value creation and delivery to customers. However, although there is significant research on digitalization and digital transformation, few studies highlight the specific needs of embedded systems companies and what it takes to transform from a traditional towards a digital company within business domains characterized by high complexity, hardware dependencies, and safety‐critical system functionality. In this paper, we capture the difference between what constitutes a traditional and a digital company and we detail the typical evolution path embedded systems companies take when transitioning towards becoming digital companies.

Frequentist statistical methods, such as hypothesis testing, are standard practice in papers that provide benchmark comparisons. Unfortunately, these methods have often been misused, e.g., without testing for their statistical test assumptions or without controlling for family-wise errors in multiple group comparisons, among several other problems. Bayesian Data Analysis (BDA) addresses many of the previously mentioned shortcomings but its use is not widely spread in the analysis of empirical data in the evolutionary computing community. This paper provides three main contributions. First, we motivate the need for utilizing Bayesian data analysis and provide an overview of this topic. Second, we discuss the practical aspects of BDA to ensure that our models are valid and the results transparent. Finally, we provide five statistical models that can be used to answer multiple research questions. The online appendix provides a step-by-step guide on how to perform the analysis of the models discussed in this paper, including the code for the statistical models, the data transformations and the discussed tables and figures.

In order to distinguish themselves from their competitors, software service providers constantly try to assess and improve customer satisfaction. However, measuring customer satisfaction in a continuous way is often time and cost intensive, or requires effort on the customer side. Especially in B2B contexts, a continuous assessment of customer satisfaction is difficult to achieve due to potential restrictions and complex provider-customer-end user setups. While concepts such as web usage mining enable software providers to get a deep understanding of how their products are used, its application to quantitatively measure customer satisfaction has not yet been studied in greater detail. For that reason, our study aims at combining existing knowledge on customer satisfaction, web usage mining, and B2B service characteristics to derive a model that enables an automated calculation of quantitative customer satisfaction scores. We apply web usage mining to validate these scores and to compare the usage behavior of satisfied and dissatisfied customers. This approach is based on domain-specific service quality and web usage metrics and is, therefore, suitable for continuous measurements without requiring active customer participation. The applicability of the model is validated by instantiating it in a real-world B2B online platform.

Machine learning and deep learning techniques are becoming increasingly popular and critical for companies as part of their systems. However, although the development and prototyping of ML/DL systems are common across companies, the transition from prototype to production-quality deployment models are challenging. One of the key challenges is how to determine the selection of an optimal architecture for AI deployment. Based on our previous research, and to offer support and guidance to practitioners, we developed a framework in which we present five architectural alternatives for AI deployment ranging from centralized to fully decentralized edge architectures. As part of our research, we validated the framework in software-intensive embedded system companies and identified key challenges they face when deploying ML/DL models. In this paper, and to further advance our research on this topic, we identify factors that help practitioners determine what architecture to select for the ML/D L model deployment. For this, we conducted a follow-up study involving interviews and workshops in seven case companies in the embedded systems domain. Based on our findings, we identify three key factors and develop a framework in which we outline how prioritization and trade-offs between these results in certain architecture. The contribution of the paper is threefold. First, we identify key factors critical for AI system deployment. Second, we present the architecture selection framework that explains how prioritization and trade-offs between key factors result in the selection of a certain architecture. Third, we discuss additional factors that mayor may not influence the selection of an optimal architecture.

Data pipelines play an important role throughout the data management process. It automates the steps ranging from data generation to data reception thereby reducing the human intervention. A failure or fault in a single step of a data pipeline has cascading effects that might result in hours of manual intervention and clean-up. Data pipeline failure due to faults at different stages of data pipelines is a common challenge that eventually leads to significant performance degradation of data-intensive systems. To ensure early detection of these faults and to increase the quality of the data products, continuous monitoring and fault detection mechanism should be included in the data pipeline. In this study, we have explored the need for incorporating automated fault detection mechanisms and mitigation strategies at different stages of the data pipeline. Further, we identified faults at different stages of the data pipeline and possible mitigation strategies that can be adopted for reducing the impact of data pipeline faults thereby improving the quality of data products. The idea of incorporating fault detection and mitigation strategies is validated by realizing a small part of the data pipeline using action research in the analytics team at a large software-intensive organization within the telecommunication domain.

Machine Learning (ML) and Artificial Intelligence (AI) have increasingly gained attention in research and industry. Federated Learning, as an approach to distributed learning, shows its potential with the increasing number of devices on the edge and the development of computing power. However, most of the current Federated Learning systems apply a single-server centralized architecture, which may cause several critical problems, such as the single-point of failure as well as scaling and performance problems. In this paper, we propose and compare four architecture alternatives for a Federated Learning system, i.e. centralized, hierarchical, regional and decentralized architectures. We conduct the study by using two well-known data sets and measuring several system performance metrics for all four alternatives. Our results suggest scenarios and use cases which are suitable for each alternative. In addition, we investigate the trade-off between communication latency, model evolution time and the model classification performance, which is crucial to applying the results into real-world industrial systems.

In recent years, the application of artificial intelligence (AI) has become an integral part of a wide range of areas, including software engineering. By analyzing various data sources generated in software engineering, it can provide valuable insights into customer behavior, product performance, bugs and errors, and many more. In practice, however, AI for software analytics and business intelligence often gets stuck in a prototypical stage and the results are rarely used to make decisions based on data. To understand the underlying root causes of this phenomenon, we conduct both an explanatory case study and a survey on the challenges of realizing and utilizing artificial intelligence in the context of software-intensive businesses. As a result, we identify a vicious circle that prevents practitioners from moving from prototypical analytics to continuous and productively usable software analytics and business intelligence based on AI.

Since the advent of mobile computing and IoT, a large amount of data is distributed around the world. Companies are increasingly experimenting with innovative ways of implementing edge/cloud (re)training of AI systems to exploit large quantities of data to optimize their business value. Despite the obvious benefits, companies face challenges as the decision on how to implement edge/cloud (re)training depends on factors such as the task intent, the amount of data needed for (re)training, edge-to-cloud data transfer, the available computing and memory resources. Based on action research in a software-intensive embedded systems company where we study multiple use cases as well as insights from our previous collaborations with industry, we develop a generic framework consisting of five architectural alternatives to deploy AI on the edge utilizing transfer learning. We validate the framework in four additional case companies and present the challenges they face in selecting the optimal architecture. The contribution of the paper is threefold. First, we develop a generic framework consisting of five architectural alternatives ranging from a centralized architecture where cloud (re)training is given priority to a decentralized architecture where edge (re)training is instead given priority. Second, we validate the framework in a qualitative interview study with four additional case companies. As an outcome of validation study, we present two variants to the architectural alternatives identified as part of the framework. Finally, we identify the key challenges that experts face in selecting an ideal architectural alternative.

Over the past 15 years, A/B testing has been a critical tool for accurate prioritization of development efforts in online and web-facing companies. As automotive companies progress on their digitalization process, A/B testing and other experimentation techniques start to be adopted. However, specific characteristics of the automotive software industry create additional challenges to the successful adoption of A/B testing. Recently, research has been conducted to investigate the challenges and opportunities for experimentation techniques in the automotive and more generally in the embedded systems domain. However, despite the collaboration with industry, previous research was based on either hypothesized or toy scenarios in companies seeking, but not yet running experimentation. Utilizing a case study method, we investigate the challenges of adopting A/B testing in two large-scale automotive companies that are currently running or preparing for their first A/B testing. The contribution of this paper is two-fold. First, we present our main findings in terms of the challenges of real A/B testing iterations in automotive vehicles. Second, we present the current, potential solutions and lessons learned from applying A/B testing in the automotive domain.

Data driven and experimental development practices provide effective means for companies to adopt a customer and market-centric way-of-working. In online companies, controlled experimentation is the primary technique to measure how customers respond to variants of deployed software. Over the recent years, and due to increasing connectivity and data collection from products in the field, these practices are being adopted also in software-intensive embedded systems companies. In these companies, experiments are run on selected instances of the system or as comparisons of previously computed data to ensure value delivery to customers, improve quality and explore new value propositions. However, to utilize the benefits of data- driven and experimental development practices, companies need to define what value factors to optimize for. For highly complex embedded systems with thousands of parameters, and with people at different levels in the organization having different opinions about the value of features, this is a challenging task. In this paper, we report on longitudinal multi-case study research in which we explore value modeling as a technique to help people in development, in product management and on the business level to align interests and agree on value factors. Based on this work, we identify five purposes of value modeling and how this technique helps accelerate critical activities in an organization. The contribution of this paper is three-fold. First, we provide empirical evidence for how value modeling is an effective technique to help companies define what to optimize for. Second, we identify five purposes of value modeling. Third, we identify the key challenges that the case companies experience when applying value modeling.

The following topics are dealt with: software development management; software quality; software engineering; software maintenance; project management; formal specification; public domain software; learning (artificial intelligence); software prototyping; software architecture.

Automatic labeling is a type of classification problem. Classification has been studied with the help of statistical methods for a long time. With the explosion of new better computer processing units (CPUs) and graphical processing units (GPUs) the interest in machine learning has grown exponentially and we can use both statistical learning algorithms as well as deep neural networks (DNNs) to solve the classification tasks. Classification is a supervised machine learning problem and there exists a large amount of methodology for performing such task. However, it is very rare in industrial applications that data is fully labeled which is why we need good methodology to obtain error-free labels. The purpose of this paper is to examine the current literature on how to perform labeling using ML, we will compare these models in terms of popularity and on what datatypes they are used on. We performed a systematic literature review of empirical studies for machine learning for labeling. We identified 43 primary studies relevant to our search. From this we were able to determine the most common machine learning models for labeling. Lack of unlabeled instances is a major problem for industry as supervised learning is the most widely used. Obtaining labels is costly in terms of labor and financial costs. Based on our findings in this review we present alternate ways for labeling data for use in supervised learning tasks.

Artificial Intelligence is becoming increasingly popular with organizations due to the success of Machine Learning and Deep Learning techniques. Using these techniques, data scientists learn from vast amounts of data to enhance behaviour in software-intensive systems. Despite the attractiveness of these techniques, however, there is a lack of systematic and structured design process for developing ML/DL models. The study uses a multiple-case study approach to explore the different activities and challenges data scientists face when developing ML/DL models in software-intensive embedded systems. In addition, we have identified seven different phases in the proposed design process leading to effective model development based on the case study. Iterations identified between phases and events which trigger these iterations optimize the design process for ML/DL models. Lessons learned from this study allow data scientists and engineers to develop high-performance ML/DL models and also bridge the gap between high demand and low supply of data scientists.

Continuous experimentation (CE) refers to a group of practices used by software companies to rapidly assess the usage, value and performance of deployed software using data collected from customers and the deployed system. Despite its increasing popularity in the development of web-facing applications, CE has not been discussed in the development process of business-to-business (B2B) mission-critical systems.

The collection of high-quality data provides a key competitive advantage to companies in their decision-making process. It helps to understand customer behavior and enables the usage and deployment of new technologies based on machine learning. However, the process from collecting the data, to clean and process it to be used by data scientists and applications is often manual, non-optimized and error-prone. This increases the time that the data takes to deliver value for the business. To reduce this time companies are looking into automation and validation of the data processes. Data processes are the operational side of data analytic workflow.

Abstract Digitalization is transforming industry to an extent that we have only seen the beginnings of. Across domains, companies experience rapid changes to their existing practices due to new technologies and new entrants that current businesses. While digitalization brings endless opportunities, it comes with challenges that require companies to strategically engage with partners in their surrounding ecosystems. In this paper, we study how companies in the embedded systems domain experience the process of transitioning from product‐based companies to businesses where software, data, and artificial intelligence (AI) play an increasingly important role. To manage this, these companies need to evolve their existing ecosystems while at the same time create new ecosystems around new technologies. This involves maintaining existing technologies such as mechanics and electronics while at the same time expanding these with software, data, and AI. We provide a strategic decision framework that helps software‐intensive embedded systems companies to successfully navigate the digital transformation. We do this in two steps. First, we present three models that provide the technical content of the strategic decision framework. Second, we provide an overview of the strategic alternatives that incumbents and new entrants have available when existing technologies are commoditizing and new technologies are introduced.

A wide variety of tools to monitor and track software systems, such as websites or smartphone applications, during runtime already exists. However, their aggregated results are often not sufficient to answer questions on a product management level since these questions address several levels of complexity and abstractions, and tend to be formulated on a rather high level, for instance concerning the efficiency of their website structure for their users. A straightforward mapping between low level metrics and high level insights is typically not possible. This causes a gap that makes it challenging to continuously provide quantitative high-level insights in real-time. In order to address this challenge, we conducted a study within three distinct platforms and products, and propose a model based on our results. After defining a case for each of the independent platforms and products, we implemented a process to measure high level insights using low level metrics for each of these cases. Next, we compared the procedures and steps that were taken in each of the cases and derived a model that describes a generic approach how to utilize and process data in order to gain higher level insights. Our model structures the steps from data to knowledge over different levels of complexity and abstraction, namely operational, tactical, and strategic. Thereby, the knowledge acquired in each phase serves as input in the next phase which increases the measurable level of complexity with each iteration. Since the steps in our model are specifically arranged as a pipeline, it enables practitioners to automate a continuous and quantitative measurement of high level insights in real-time.

Deep learning is one of the most exciting and fast-growing techniques in Artificial Intelligence. The unique capacity of deep learning models to automatically learn patterns from the data differentiates it from other machine learning techniques. Deep learning is responsible for a significant number of recent breakthroughs in AI. However, deep learning models are highly dependent on the underlying data. So, consistency, accuracy, and completeness of data is essential for a deep learning model. Thus, data management principles and practices need to be adopted throughout the development process of deep learning models. The objective of this study is to identify and categorise data management challenges faced by practitioners in different stages of end-to-end development. In this paper, a case study approach is employed to explore the data management issues faced by practitioners across various domains when they use real-world data for training and deploying deep learning models. Our case study is intended to provide valuable insights to the deep learning community as well as for data scientists to guide discussion and future research in applied deep learning with real-world data.

Abstract The shift from sequential to agile software development originates from relatively small and co-located teams but soon gained prominence in larger organizations. How to apply and scale agile practices to fit the needs of larger projects has been studied to quite an extent in previous research. However, scaling agile beyond organizational boundaries, for instance in a software ecosystem context, raises additional challenges that existing studies and approaches do not yet investigate or address in great detail. For that reason, we conducted a case study in two software ecosystems that comprise several agile actors from different organizations and, thereby, scale development across organizational boundaries, in order to elaborate and understand their coordination challenges. Our results indicate that most of the identified challenges are caused by long communication paths and a lack of established processes to facilitate these paths. As a result, the participants in our study, among others, experience insufficient responsivity, insufficient communication of prioritizations and deliverables, and alterations or loss of information. As a consequence, agile practices need to be extended to fit the identified needs.

Abstract Artificial intelligence enabled systems have been an inevitable part of everyday life. However, efficient software engineering principles and processes need to be considered and extended when developing AI- enabled systems. The objective of this study is to identify and classify software engineering challenges that are faced by different companies when developing software-intensive systems that incorporate machine learning components. Using case study approach, we explored the development of machine learning systems from six different companies across various domains and identified main software engineering challenges. The challenges are mapped into a proposed taxonomy that depicts the evolution of use of ML components in software-intensive system in industrial settings. Our study provides insights to software engineering community and research to guide discussions and future research into applied machine learning.

Online Controlled Experiments (OCEs) are the norm in data-driven software companies because of the benefits they provide for building and deploying software. Product teams experiment to accurately learn whether the changes that they do to their products (e.g. adding new features) cause any impact (e.g. customers use them more frequently). Experiments also help reduce the risk from deploying software by minimizing the magnitude and duration of harm caused by software bugs, allowing software to be shipped more frequently. To make informed decisions in product development, experiment analysis needs to be granular with a large number of metrics over heterogeneous devices and audiences. Discovering experiment insights by hand, however, can be cumbersome. In this paper, and based on case study research at a large-scale software development company with a long tradition of experimentation, we (1) describe the standard process of experiment analysis, and (2) introduce an artifact to improve the effectiveness and comprehensiveness of this process.

Online Controlled Experiments (OCEs, aka A/B tests) are one of the most powerful methods for measuring how much value new features and changes deployed to software products bring to users. Companies like Microsoft, Amazon, and Booking.com report the ability to conduct thousands of OCEs every year. However, the competences of the remainder of the online software industry remain unknown. The main objective of this paper is to reveal the current state of A/B testing maturity in the software industry based on a maturity model from our previous research. We base our findings on 44 responses from an online empirical survey. Our main contribution of this paper is the current state of experimentation maturity as operationalized by the ExG model for a convenience sample of companies doing online controlled experiments. Our findings show that, among others, companies typically develop in-house experimentation platforms, that these platforms are of various levels of maturity, and that designing key metrics - Overall Evaluation Criteria - remains the key challenge for successful experimentation.

Empowerment is based on the belief that employees have the ability, and the desire, to shoulder more responsibility and perform better when given freedom. In an empowered organization, authority is given to employees with the intent to increase responsiveness to customers, improve decision-making power and to increase team motivation and skills. However, while most studies picture empowerment as the "ideal state" and the place where all organizations strive to be, our research shows that fully empowered teams without strategic guidance suffer from a number of problems. Based on multi-case study research in eleven software-intensive companies, we see that companies need to allow for different levels of empowerment depending on what they aim to achieve, characteristics of the industry domain, the business model and other factors, and that strategic guidance is critical to set direction and for avoiding chaos. To help companies approach the optimal level of empowerment, we provide a framework consisting of two inter-connected models that help companies to, rather than staying in their current hierarchical structures, transition to a level of empowerment that maximizes business value and performance.

Online controlled experiments are extensively used by web-facing companies to validate and optimize their systems, providing a competitive advantage in their business. As the number of experiments scale, companies aim to invest their experimentation resources in larger feature changes and leave the automated techniques to optimize smaller features. Optimization experiments in the continuous space are encompassed in the many-armed bandits class of problems. Although previous research provides algorithms for solving this class of problems, these algorithms were not implemented in real-world online experimentation problems and do not consider the application constraints, such as time to compute a solution, selection of a best arm and the estimation of the mean-reward function. This work discusses the online experiments in context of the many-armed bandits class of problems and provides three main contributions: (1) an algorithm modification to include online experiments constraints, (2) implementation of this algorithm in an industrial setting in collaboration with Sony Mobile, and (3) statistical evidence that supports the modification of the algorithm for online experiments scenarios. These contributions support the relevance of the LG-HOO algorithm in the context of optimization experiments and show how the algorithm can be used to support continuous optimization of online systems in stochastic scenarios.

Abstract Context: Continuous experimentation is frequently used in web-facing companies and it is starting to gain the attention of embedded systems companies. However, embedded systems companies have different challenges and requirements to run experiments in their systems. Objective: This paper explores the challenges during the adoption of continuous experimentation in embedded systems from both industry practice and academic research. It presents strategies, guidelines, and solutions to overcome each of the identified challenges. Method: This research was conducted in two parts. The first part is a literature review with the aim to analyze the challenges in adopting continuous experimentation from the research perspective. The second part is a multiple case study based on interviews and workshop sessions with five companies to understand the challenges from the industry perspective and how they are working to overcome them. Results: This study found a set of twelve challenges divided into three areas; technical, business, and organizational challenges and strategies grouped into three categories, architecture, data handling and development processes. Conclusions: The set of identified challenges are presented with a set of strategies, guidelines, and solutions. To the knowledge of the authors, this paper is the first to provide an extensive list of challenges and strategies for continuous experimentation in embedded systems. Moreover, this research points out open challenges and the need for new tools and novel solutions for the further development of experimentation in embedded systems.

Abstract Today, companies operate in business ecosystems where they collaborate, compete, share, and learn from others with benefits such as to present more attractive offerings and sharing innovation costs. With ecosystems being the new way of operating, the ability to strategically reposition oneself to increase or shift power balance is becoming key for competitive advantage. However, companies run into a number of traps when trying to realize strategical changes in their ecosystems. In this paper, we identify 5 traps that companies fall into. First, the “descriptive versus prescriptive trap” is when companies assume that current boundaries between partners are immutable. Second, the “assumptions trap” is when powerful ecosystem partners assume that they understand what others regard as value‐adding without validating their assumptions. Third, the “keeping it too simple trap” is when companies overlooks the effort required to align interests. Fourth, the “doing it all at once trap” is when companies disrupt an ecosystem assuming that all partners can change direction at the same time. Finally, the “planning trap” is when companies are unable to move forward without a complete plan. We provide empirical evidence for each trap, and we propose an ecosystem engagement process for how to avoid falling into these.

Abstract Companies need to know how much value their ideas deliver to customers. One of the most powerful ways to accurately measure this is by conducting online controlled experiments (OCEs). To run experiments, however, companies need to develop strong experimentation practices as well as align their organization and culture to experimentation. The main objective of this paper is to demonstrate how to run OCEs at large scale using the experience of companies that succeeded in scaling. Based on case study research at Microsoft, Booking.com, Skyscanner, and Intuit, we present our main contribution—The Experiment Growth Model. This four‐stage model addresses the seven critical aspects of experimentation and can help companies to transform their organizations into learning laboratories where new ideas can be tested with scientific accuracy. Ultimately, this should lead to better products and services.

Online controlled experiments (for example A/B tests) are increasingly being performed to guide product development and accelerate innovation in online software product companies. The benefits of controlled experiments have been shown in many cases with incremental product improvement as the objective. In this paper, we demonstrate that the value of controlled experimentation at scale extends beyond this recognized scenario. Based on an exhaustive and collaborative case study in a large software-intensive company with highly developed experimentation culture, we inductively derive the benefits of controlled experimentation. The contribution of our paper is twofold. First, we present a comprehensive list of benefits and illustrate our findings with five case examples of controlled experiments conducted at Microsoft. Second, we provide guidance on how to achieve each of the benefits. With our work, we aim to provide practitioners in the online domain with knowledge on how to use controlled experimentation to maximize the benefits on the portfolio, product and team level.

Innovation and optimization in software systems can occur from pre-development to post-deployment stages. Companies are increasingly reporting the use of experiments with customers in their systems in the post-deployment stage. Experiments with customers and users are can lead to a significant learning and return-on-investment. Experiments are used for both validation of manual hypothesis testing and feature optimization, linked to business goals. Automated experimentation refers to having the system controlling and running the experiments, opposed to having the R&D organization in control. Currently, there are no systematic approaches that combine manual hypothesis validation and optimization in automated experiments. This paper presents concepts related to automated experimentation, as controlled experiments, machine learning and software architectures for adaptation. However, this paper focuses on how architectural aspects that can contribute to support automated experimentation. A case study using an autonomous system is used to demonstrate the developed initial architecture framework. The contributions of this paper are threefold. First, it identifies software architecture qualities to support automated experimentation. Second, it develops an initial architecture framework that supports automated experiments and validates the framework with an autonomous mobile robot. Third, it identifies key research challenges that need to be addressed to support further development of automated experimentation.

Software development companies are increasingly aiming to become data-driven by trying to continuously experiment with the products used by their customers. Although familiar with the competitive edge that the A/B testing technology delivers, they seldom succeed in evolving and adopting the methodology. In this paper, and based on an exhaustive and collaborative case study research in a large software-intense company with highly developed experimentation culture, we present the evolution process of moving from ad-hoc customer data analysis towards continuous controlled experimentation at scale. Our main contribution is the "Experimentation Evolution Model" in which we detail three phases of evolution: technical, organizational and business evolution. With our contribution, we aim to provide guidance to practitioners on how to develop and scale continuous experimentation in software organizations with the purpose of becoming data-driven at scale.

Recently, business ecosystems have been recognized as one of the most interesting phenomenon in software engineering research. Companies experience a paradigm shift where product development and innovation is moving outside the boundaries of the firm and where networks of stakeholders join forces to co-create value. While there is prominent research focusing on the managerial perspective of business ecosystems, few studies provide strategic guidance for how to intentionally manage the different ecosystems that companies operate in. Therefore, and on the basis of multicase study research, we provide empirical evidence on the challenges that software-intensive companies experience in relation to the different types of business ecosystems they operate in. We conduct a “state-of-the-art” literature review to identify strategies that are used to manage ecosystem engagements, and we develop a conceptual model in which we identify strategies for managing the innovation ecosystem, the differentiating ecosystem, and the commoditizing ecosystem. By categorising the different strategies in relation to the different types of ecosystems for which they are valid, the “three-layer ecosystem strategy model” provides comprehensive support for strategy selection. We validate the use of the identified strategies in 6 software-intensive case companies, and we provide empirical insights on the “relevance” and the “desired use” of these strategies as experienced by the case companies.

With continuous deployment of software functionality, a constant flow of new features to products is enabled. Although new functionality has potential to deliver improvements and possibilities that were previously not available, it does not necessary generate business value. On the contrary, with fast and increasing system complexity that is associated with high operational costs, more waste than value risks to be created. Validating how much value a feature actually delivers, project how this value will change over time, and know when to remove the feature from the product are the challenges large software companies increasingly experience today. We propose and study the concept of a software feature lifecycle from a value point of view, i.e. how companies track feature value throughout the feature lifecycle. The contribution of this paper is a model that illustrates how to determine (1) when to add the feature to a product, (2) how to track and (3) project the value of the feature during the lifecycle, and how to (4) identify when a feature is obsolete and should be removed from the product.

Traditionally, innovation initiatives in software-intensive systems companies are viewed as either internal innovation, such as technology driven innovation based on ideas generated within a company, as collaborative innovation where a number of stakeholders co-create value, or as external innovation in which companies adopt strategies to capture and expand on ideas created by other stakeholders. However, and based on longitudinal case study research in six software-intense companies in the embedded systems domain, we see that most innovation strategies involve a mix of internal, collaborative and external elements. Due to the dichotomy in approaches however, companies often fail to select the optimal innovation strategy for the specific innovation challenge at hand. As a result, innovation initiatives suffer and companies and their ecosystem partners cannot fully capitalize on the value created. In this paper, we present a conceptual framework in which we identify twelve different ecosystem-centric innovation strategies. For each strategy, we identify the internal, the collaborative and the external elements. Also, and based on our empirical findings, we provide guidelines on the optimal selection of strategies.

With agile teams becoming increasingly multi-disciplinary and including all functions, the role of customer feedback is gaining momentum. Today, companies collect feedback directly from customers, as well as indirectly from their products. As a result, companies face a situation in which the amount of data from which they can learn about their customers is larger than ever before. In previous studies, the collection of data is often identified as challenging. However, and as illustrated in our research, the challenge is not the collection of data but rather how to share this data among people in order to make effective use of it. In this paper, and based on case study research in three large software-intensive companies, we (1) provide empirical evidence that ‘lack of sharing’ is the primary reason for insufficient use of customer and product data, and (2) develop a model in which we identify what data is collected, by whom data is collected and in what development phases it is used. In particular, the model depicts critical hand-overs where certain types of data get lost, as well as the implications associated with this. We conclude that companies benefit from a very limited part of the data they collect, and that lack of sharing of data drives inaccurate assumptions of what constitutes customer value.

Visual GUI testing (VGT) is an emerging technique that provides software companies with the capability to automate previously time-consuming, tedious, and fault prone manual system and acceptance tests. Previous work on VGT has shown that the technique is industrially applicable, but has not addressed the real-world applicability of the technique when used by practitioners on industrial grade systems. This paper presents a case study performed during an industrial project with the goal to transition from manual to automated system testing using VGT. Results of the study show that the VGT transition was successful and that VGT could be applied in the industrial context when performed by practitioners but that there were several problems that first had to be solved, e.g. testing of a distributed system, tool volatility. These problems and solutions have been presented together with qualitative, and quantitative, data about the benefits of the technique compared to manual testing, e.g. greatly improved execution speed, feasible transition and maintenance costs, improved bug finding ability. The study thereby provides valuable, and previously missing, contributions about VGT to both practitioners and researchers.

System- and acceptance-testing are primarily performed with manual practices in current software industry. However, these practices have several issues, e.g. they are tedious, error prone and time consuming with costs up towards 40 percent of the total development cost. Automated test techniques have been proposed as a solution to mitigate these issues, but they generally approach testing from a lower level of system abstraction, leaving a gap for a flexible, high system-level test automation technique/tool. In this paper we present JAutomate, a Visual GUI Testing (VGT) tool that fills this gap by combining image recognition with record and replay functionality for high system-level test automation performed through the system under test's graphical user interface. We present the tool, its benefits compared to other similar techniques and manual testing. In addition, we compare JAutomate with two other VGT tools based on their static properties. Finally, we present the results from a survey with industrial practitioners that identifies test-related problems that industry is currently facing and discuss how JAutomate can solve or mitigate these problems.

In existing global software development (GSD) literature, much focus has been on identifying the challenges that practitioners may face (such as socio-cultural and temporal distance issues), while potential benefits have not been extensively analyzed. We reverse this trend by studying these potential benefits. We question whether they are well-founded assumptions and whether they are attainable in practice. This paper presents findings from a multi-case study at three multi-national companies that have extensive experience in GSD. We identify the benefits mentioned in GSD literature, analyze them with regards to the companies' experiences and then conclude whether or not each benefit is being realized in practice. Our findings reveal that the realization of the assumed benefits cannot be simply taken for granted

Global software development (GSD) is a phenomenon that is receiving considerable interest from companies all over the world. In GSD, stakeholders from different national and organizational cultures are involved in developing software and the many benefits include access to a large labour pool, cost advantage and round-the-clock development. However, GSD is technologically and organizationally complex and presents a variety of challenges to be managed by the software development team. In particular, temporal, geographical and socio-cultural distances impose problems not experienced in traditional systems development. In this paper, we present findings from a case study in which we explore the particular challenges associated with managing GSD. Our study also reveals some of the solutions that are used to deal with these challenges. We do so by empirical investigation at three US based GSD companies operating in Ireland. Based on qualitative interviews we present challenges related to temporal, geographical and socio-cultural distance