Randomised field experiments, such as A/B testing, have long been the gold standard for evaluating the value that new software brings to customers. However, running randomised field experiments is not always desired, possible or even ethical in the development of automotive embedded software. In the face of such restrictions, we propose the use of the Bayesian propensity score matching technique for causal inference of observational studies in the automotive domain. In this paper, we present a method based on the Bayesian propensity score matching framework, applied in the unique setting of automotive software engineering. This method is used to generate balanced control and treatment groups from an observational online evaluation and estimate causal treatment effects from the software changes, even with limited samples in the treatment group. We exemplify the method with a proof-of-concept in the automotive domain. In the example, we have a larger control (Nc = 1100) fleet of cars using the current software and a small treatment fleet (Nt = 38), in which we introduce a new software variant. We demonstrate a scenario that shipping of a new software to all users is restricted, as a result, a fully randomised experiment could not be conducted. Therefore, we utilised the Bayesian propensity score matching method with 14 observed covariates as inputs. The results show more balanced groups, suitable for estimating causal treatment effects from the collected observational data. We describe the method in detail and share our configuration. Furthermore, we discuss how can such a method be used for online evaluation of new software utilising small groups of samples.

A/B experimentation is a known technique for data-driven product development and has demonstrated its value in web-facing businesses. With the digitalisation of the automotive industry, the focus in the industry is shifting towards software. For automotive embedded software to continuously improve, A/B experimentation is considered an important technique. However, the adoption of such a technique is not without challenge. In this paper, we present an architecture to enable A/B testing in automotive embedded software. The design addresses challenges that are unique to the automotive industry in a systematic fashion. Going from hypothesis to practice, our architecture was also applied in practice for running online experiments on a considerable scale. Furthermore, a case study approach was used to compare our proposal with state-of-practice in the automotive industry. We found our architecture design to be relevant and applicable in the efforts of adopting continuous A/B experiments in automotive embedded software.

A/B testing is gaining attention in the automotive sector as a promising tool to measure causal effects from software changes. Different from the web-facing businesses, where A/B testing has been well-established, the automotive domain often suffers from limited eligible users to participate in online experiments. To address this shortcoming, we present a method for designing balanced control and treatment groups so that sound conclusions can be drawn from experiments with considerably small sample sizes. While the Balance Match Weighted method has been used in other domains such as medicine, this is the first paper to apply and evaluate it in the context of software development. Furthermore, we describe the Balance Match Weighted method in detail and we conduct a case study together with an automotive manufacturer to apply the group design method in a fleet of vehicles. Finally, we present our case study in the automotive software engineering domain, as well as a discussion on the benefits and limitations of the A/B group design method.

Over the past 15 years, A/B testing has been a critical tool for accurate prioritization of development efforts in online and web-facing companies. As automotive companies progress on their digitalization process, A/B testing and other experimentation techniques start to be adopted. However, specific characteristics of the automotive software industry create additional challenges to the successful adoption of A/B testing. Recently, research has been conducted to investigate the challenges and opportunities for experimentation techniques in the automotive and more generally in the embedded systems domain. However, despite the collaboration with industry, previous research was based on either hypothesized or toy scenarios in companies seeking, but not yet running experimentation. Utilizing a case study method, we investigate the challenges of adopting A/B testing in two large-scale automotive companies that are currently running or preparing for their first A/B testing. The contribution of this paper is two-fold. First, we present our main findings in terms of the challenges of real A/B testing iterations in automotive vehicles. Second, we present the current, potential solutions and lessons learned from applying A/B testing in the automotive domain.

To investigate the new requirements and challenges of architecting often safety critical software in the automotive domain, we have performed two case studies on Volvo Car Group and Volvo Group Truck Technology. Our findings suggest that automotive software architects produce two different architectures (or views) of the same system. The first one is a high-level descriptive architecture, mainly documenting system design decisions and describing principles and guidelines that should govern the overall system. The second architecture is the working architecture, defining the actual blueprint for the implementation teams and being used in their daily work. The working architecture is characterized by high complexity and considerably lower readability than the high-level architecture. Unfortunately, the team responsible for the high-level architecture tends to get isolated from the rest of the development organization, with few communications except regarding the working architecture. This creates tensions within the organizations, sub-optimal design of the communication matrix and limited usage of the high-level architecture in the development teams. To adapt to the current pace of software development and rapidly growing software systems new ways of working are required, both on technical and on an organizational level.

R&D at Volvo Car Group (VCG) has made great investments during the last decades in Model Based Development and Physical Modelling of the complex mechatronic systems associated with vehicles. Driven by the dramatic increase of electronics and software in competitive cars, enhanced also by the novel development of hybrid vehicles, the recent focus has been to scale agile mechatronic software development in the multi ECU system, utilizing the tools of and experience in modelling. VCG is currently developing ECU software using executable Simulink models with automated code generation and is currently in the phase of developing the essential automation frameworks for fast "continuous" integration and testing. Early integration and fast development loops are important but challenging in vehicle systems, while multiple suppliers as well as in-house developers are involved and where advanced control software development is conducted by domain experts, not software specialists. In parallel, much effort is spent in development of Plant models (usually physical models of dependent systems or mechanics). Plant models can be used both for development, which is usually equivalent with Understanding the system, and for various software regression tests. Case studies made at VCG shows that modelling should actually be considered as an enabler for agile development. The current challenges include efficient test environments for developers, fast and automated feedback from system or sub system integration (real and virtual), scaling of plant model architectures and finally the challenge of controlling the potentially diverging amount of variants, in mechatronics, software and models.