This paper provides a progress update to the doctoral symposium paper accepted at CAIN 2024 [1], which introduced the Experimentation Engine for optimizing data analytics workflows through user-driven experimentation. Over the past year, the framework has been significantly refined to better address diverse user needs by adapting the experimentation process for different user types, including domain experts and data scientists. The updated Experimentation Engine emphasizes continuous user involvement throughout the experimentation process, facilitating seamless interaction and iterative refinement. A demonstration video of the initial prototype, showcasing its capabilities, is available at https://doi.org/10.5281/zenodo.14288226. These enhancements aim to further streamline the experimentation process, providing accurate, meaningful, and trustworthy results while dynamically adapting to varying user requirements and system constraints. The key updates and lessons learned during this process are discussed in this paper.

Despite advancements in MLOps and AutoML, ML development still remains challenging for data scientists. First, there is poor support for and limited control over optimizing and evolving ML models. Second, there is lack of efficient mechanisms for continuous evolution of ML models which would leverage the knowledge gained in previous optimizations of the same or different models. We propose an experiment-driven MLOps approach which tackles these problems. Our approach relies on the concept of an experiment, which embodies a fully controllable optimization process. It introduces full traceability and repeatability to the optimization process, allows humans to be in full control of it, and enables continuous improvement of the ML system. Importantly, it also establishes knowledge, which is carried over and built across a series of experiments and allows for improving the efficiency of experimentation over time. We demonstrate our approach through its realization and application in the ExtremeXp11https://extremexp.eu/ project (Horizon Europe).

Machine Learning (ML) has advanced significantly, yet the development of ML workflows still relies heavily on expert intuition, limiting standardization. MLOps integrates ML workflows for reliability, while AutoML automates tasks like hyperparameter tuning. However, these approaches often overlook the iterative and experimental nature of the development of ML workflows. Within the ongoing ExtremeXP project (Horizon Europe), we propose an experiment-driven approach where systematic experimentation becomes central to ML workflow evolution. The framework created within the project supports transparent, reproducible, and adaptive experimentation through a formal metamodel and related domain-specific language. Key principles include traceable experiments for transparency, empowered decision-making for data scientists, and adaptive evolution through continuous feedback. In this paper, we present the framework from the model-based approach perspective. We discuss the lessons learned from the use of the metamodel-centric approach within the project—especially with use-case partners without prior modeling expertise.

In the Big Data era, efficient data analytics workflows are imperative to extract useful and meaningful insights. Data analysts and scientists spend an inordinate amount of time finding the best workflow via trial and error to get accurate and meaningful results that meet their expectations. We propose an Experimentation Engine that selects and optimizes the best workflow variant through continuous experimentation and having the user in the loop. Experimentation Engine saves time finding the workflow that satisfies the user requirements and provides accurate, useful and trustworthy results.