Abstract Experimentation has become a cornerstone of agile, evidence-driven software development. To explore how it may evolve in the coming decade, we collected qualitative data from 58 experts across the global experimentation community. Using an inductive thematic analysis inspired by the Gioia methodology, we identified six interrelated trends that capture how practitioners envision the future of experimentation: AI-augmented workflows, segment-level personalization, platformization and warehouse-native architectures, expansion beyond web contexts, rigor at scale, and cultural capability building. These trends highlight experimentation’s evolution from a technical testing practice toward a socio-technical learning system. Building on these insights, the paper outlines a practice-inspired research agenda for the next decade of evidence-based product development.

Abstract Experimentation programs are vital for enabling data-driven decision-making within product development. However, evaluating their overarching success remains a significant challenge. Current metrics, such as conversion rates, primarily focus on individual experiments, leaving a gap in assessing broader program efficiency and impact. This paper addresses this gap by presenting a structured overview and analysis of 18 program-level metrics, categorized into six domains: Volume, Outcome-Based, Quality, Engagement, Process Efficiency and Strategic Alignment. Metrics such as experimentation throughput, time-to-decision and experimentation coverage are examined for their implications on operational efficiency, cultural adoption, and strategic alignment. Based on interviews with 48 experimentation practitioners, this work provides a description of these metrics and discusses their benefits and challenges. The results offer actionable insights for advancing experimentation practices and aligning them with organizational goals.

We represent a vehicle dynamics model for autonomous driving near the limits of handling via a multilayer neural network. Online adaptation is desirable in order to address unseen environments. However, the model needs to adapt to new environments without forgetting previously encountered ones. In this study, we apply Continual-MAML to overcome this difficulty. It enables the model to adapt to the previously encountered environments quickly and efficiently by starting updates from optimized initial parameters. We evaluate the impact of online model adaptation with respect to inference performance and impact on control performance of a model predictive path integral (MPPI) controller using the TRIKart platform. The neural network was pre-trained using driving data collected in our test environment, and experiments for online adaptation were executed on multiple different road conditions not contained in the training data. Empirical results show that the model using Continual-MAML outperforms the fixed model and the model using gradient descent in test set loss and online tracking performance of MPPI.

Game theoretic methods have become popular for planning and prediction in situations involving rich multi-agent interactions. However, these methods often assume the existence of a single local Nash equilibria and are hence unable to handle uncertainty in the intentions of different agents. While maximum entropy (MaxEnt) dynamic games try to address this issue, practical approaches solve for MaxEnt Nash equilibria using linear-quadratic approximations which are restricted to unimodal responses and unsuitable for scenarios with multiple local Nash equilibria. By reformulating the problem as a POMDP, we propose MPOGames, a method for efficiently solving MaxEnt dynamic games that captures the interactions between local Nash equilibria. We show the importance of uncertainty-aware game theoretic methods via a two-agent merge case study. Finally, we prove the real-time capabilities of our approach with hardware experiments on a 1/10th scale car platform.

Initiatives such as the 2020 IEEE Global Road Damage Detection Challenge prompted extensive research in camera-based road damage detection with Deep Learning, primarily focused on improving the efficiency of road management. However, road damage detection is also relevant for automated driving to optimize passenger comfort and safety. We use the state-of-the-art object detection framework Scaled-YOLOv4 and develop two small-sized models that cope with the limited computational resources in the vehicle. With average F1 scores of 0.54 and 0.586, respectively, the models keep pace with the state-of-the-art solutions of the challenge. Since the data consists only of smartphone images, we also train expert models for autonomous driving utilizing vehicle camera data. In addition to detection, severity assessment is critical. We propose a semi-supervised learning approach based on the encodings learned by combining a class-conditional Variational Autoencoder and a Wasserstein Generative Adversarial Network to classify detected damage into different severity levels.

Road damages are of great interest for federal road authorities and their infrastructure management as well as the automated driving task and thus safety and comfort of vehicle occupants. Therefore, we are investigating the automatic detection of different types of road damages by images from a front-facing camera in the vehicle. The data basis of our work is provided by the ’IEEE BigData Cup Challenge’ and its dataset ’RDD-2020’ with a large number of labelled images from Japan, India and the Czech Republic. Our Deep Learning approach utilizes the pre-trained Faster Region Based Convolutional Neural Networks (R-CNN). In the first step, we classify the destination of the image followed by expert networks for each region. Between the explanation of our applied Deep Learning methodology, some remaining sources of errors are discussed and further, partly failed approaches during our development period are displayed, which could be of interest for future work. Our results are convincing and we are able to achieve an F1 score of 0.487 across all regions for longitudinal and lateral cracks, alligator cracks and potholes.

In the past years, automated driving has become one of the most important research fields in the automotive industry. A key component for a successful substitution of human driving by vehicles is a real-time model of the current environment including the traffic situation, the guide-way, and the road itself. Although, most of the information for the environment model are provided via in-vehicle generated data based on camera, LIDAR, and RADAR sensors, we propose a solution of classifying road quality within the spring-damper system of the vehicle. In this paper, we utilize the Vehicle Level Sensor (VLS), which is a standard component in modern vehicles, for road condition assessment. We present a simulation of the Quarter Vehicle Model (QVM) for road elevation measurement to enable each connected vehicle to provide valid data for a potential crowd sensing approach where every vehicle contributes data for past and consumes data for upcoming segments. The generated data is capable of providing the environment model with real-time data of upcoming road segments. The simulation results are validated on a test bench including a review of the errors.

We introduce an algorithm for autonomous control of multiple fast ground vehicles operating in close proximity to each other. The algorithm is based on a combination of the game theoretic notion of iterated best response, and an information theoretic model predictive control algorithm designed for non-linear stochastic systems. We test the algorithm on two one-fifth scale AutoRally platforms traveling at speeds upwards of 8 meters per second, while maintaining a following distance of under two meters from bumper-to-bumper.

We present an information-theoretic approach to stochastic optimal control problems that can be used to derive general sampling-based optimization schemes. This new mathematical method is used to develop a sampling-based model predictive control algorithm. We apply this information-theoretic model predictive control scheme to the task of aggressive autonomous driving around a dirt test track, and compare its performance with a model predictive control version of the cross-entropy method.

We introduce an information theoretic model predictive control (MPC) algorithm capable of handling complex cost criteria and general nonlinear dynamics. The generality of the approach makes it possible to use multi-layer neural networks as dynamics models, which we incorporate into our MPC algorithm in order to solve model-based reinforcement learning tasks. We test the algorithm in simulation on a cart-pole swing up and quadrotor navigation task, as well as on actual hardware in an aggressive driving task. Empirical results demonstrate that the algorithm is capable of achieving a high level of performance and does so only utilizing data collected from the system.

In this paper we present a model predictive control algorithm designed for optimizing non-linear systems subject to complex cost criteria. The algorithm is based on a stochastic optimal control framework using a fundamental relationship between the information theoretic notions of free energy and relative entropy. The optimal controls in this setting take the form of a path integral, which we approximate using an efficient importance sampling scheme. We experimentally verify the algorithm by implementing it on a Graphics Processing Unit (GPU) and apply it to the problem of controlling a fifth-scale Auto-Rally vehicle in an aggressive driving task.