Despite the increasing adoption of Machine Learning Operations (MLOps), teams still encounter challenges in effectively applying this paradigm to their projects. While numerous MLOps tools exist, consolidated knowledge to inform architecture design is still lacking. In response, our goal is to provide a comprehensive overview of MLOps architectures from a structural and process perspective, the tools mentioned supporting the implementation of architecture components, and the stakeholders responsible for the MLOps process. We conduct a systematic mapping study of 93 primary studies to collect and analyze the state of the art knowledge on MLOps systems using automatic, manual, and snowballing-based search strategies. Subsequently, we use card sorting to synthesize the results. We contribute: (i) a categorization of 39 MLOps architecture components and a description of several MLOps architecture variants; (ii) a systematic map between the components and the existing MLOps tools; (iii) a description of 56 process steps for MLOps systems creation, deployment, and maintenance; and (iv) a description of MLOps stakeholders, their responsibilities, and a systematic map between the process steps and the responsible stakeholders. Our results serve as an overview of the state of the art in MLOps architectures from a structural and a process perspective to support researchers and practitioners in the architecture design of their MLOps systems.

Machine learning (ML) models are often integrated into ML-enabled systems to provide software functionality that would otherwise be impossible. This integration requires the selection of an appropriate ML model export format, for which many options are available. These formats are crucial for ensuring a seamless integration, and choosing a suboptimal one can negatively impact system development, e.g., via increased dependencies and higher maintenance costs. However, little evidence is available to guide practitioners during the export format selection. We therefore aim to comprehensively evaluate various model export formats regarding their impact on the development of ML-enabled systems from an integration perspective. Based on the results of a preliminary questionnaire survey (n=17), we designed an extensive embedded case study with two ML-enabled systems in three versions with different technologies. We then analyzed the effect of five popular export formats, namely ONNX, Pickle, TensorFlow's SavedModel, PyTorch's TorchScript, and Joblib. In total, we studied 30 units of analysis (2 systems x 3 tech stacks x 5 formats) and collected data via structured field notes. The holistic qualitative analysis of the results indicated that ONNX offered the most efficient integration and portability across most cases. SavedModel and TorchScript were very convenient to use in Python-based systems, but otherwise required workarounds (TorchScript more than SavedModel). SavedModel also allowed the easy incorporation of preprocessing logic into a single file, which made it scalable for complex deep learning use cases. Pickle and Joblib were the most challenging to integrate, even in Python-based systems. Regarding technical support, all model export formats had strong technical documentation and strong community support across platforms such as Stack Overflow and Reddit. Practitioners can use our findings to inform the selection of ML export formats suited to their context.

Despite advancements in MLOps and AutoML, ML development still remains challenging for data scientists. First, there is poor support for and limited control over optimizing and evolving ML models. Second, there is lack of efficient mechanisms for continuous evolution of ML models which would leverage the knowledge gained in previous optimizations of the same or different models. We propose an experiment-driven MLOps approach which tackles these problems. Our approach relies on the concept of an experiment, which embodies a fully controllable optimization process. It introduces full traceability and repeatability to the optimization process, allows humans to be in full control of it, and enables continuous improvement of the ML system. Importantly, it also establishes knowledge, which is carried over and built across a series of experiments and allows for improving the efficiency of experimentation over time. We demonstrate our approach through its realization and application in the ExtremeXp11https://extremexp.eu/ project (Horizon Europe).

Machine Learning (ML) has advanced significantly, yet the development of ML workflows still relies heavily on expert intuition, limiting standardization. MLOps integrates ML workflows for reliability, while AutoML automates tasks like hyperparameter tuning. However, these approaches often overlook the iterative and experimental nature of the development of ML workflows. Within the ongoing ExtremeXP project (Horizon Europe), we propose an experiment-driven approach where systematic experimentation becomes central to ML workflow evolution. The framework created within the project supports transparent, reproducible, and adaptive experimentation through a formal metamodel and related domain-specific language. Key principles include traceable experiments for transparency, empowered decision-making for data scientists, and adaptive evolution through continuous feedback. In this paper, we present the framework from the model-based approach perspective. We discuss the lessons learned from the use of the metamodel-centric approach within the project—especially with use-case partners without prior modeling expertise.

Context: Robotics software architecture-based self-adaptive systems (RSASSs) are robotics systems made robust to runtime uncertainty by adapting their software architectures. The research landscape of RSASS approaches is multidisciplinary and fragmented, with many aspects still unexplored or ineffectively shared among communities involved. Objective: We aim at identifying, classifying, and analyzing the state of the art of existing approaches for RSASSs from the following perspectives: (i) the key characteristics of approaches and (ii) the evaluation strategies applied by researchers. Method: We apply the systematic mapping research method. We selected 37 primary studies via automatic, manual, and snowballing-based search and selection procedures. We rigorously defined and applied a classification framework composed of 32 parameters and synthesize the obtained data to produce a comprehensive overview of the state of the art. Results: This work contributes (i) a rigorously defined classification framework for studies on RSASSs, (ii) a systematic map of the research efforts on RSASSs, (iii) a discussion of emerging findings and implications for future research, and (iv) a publicly available replication package. Conclusion: This study provides a solid evidence-based overview of the state of the art in RSASS approaches. Its results can benefit RSASS researchers at different levels of seniority and involvement in RSASS research. Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board .

Machine learning (ML) models in production fail when their broader systems – from data pipelines to deployment environments – deviate from training assumptions, not merely due to statistical anomalies in input data. Despite extensive work on data drift, data validation, and out-of-distribution detection, ML monitoring research remains largely model-centric while neglectingcontextual information: auxiliary signals about the system around the model (external factors, data pipelines, downstream applications). Incorporating this context turns statistical anomalies into actionable alerts and structured root-cause analysis. Drawing on a systematic review of 94 primary studies, we identify three dimensions of contextual information for ML monitoring: thesystemelement concerned (natural environment or technical infrastructure); theaspectof that element (runtime states, structural relationships, prescriptive properties); and therepresentationused (formal constructs or informal formats). This forms theContextual System-Aspect-Representation(C-SAR) framework, a descriptive model synthesizing our findings. We identify 20 recurring triplets across these dimensions and map them to the monitoring activities they support. This study provides a holistic perspective on ML monitoring: from interpreting “tea leaves” (i.e., isolated data and performance statistics) to constructing and managing “system maps” (i.e., end-to-end views that connect data, models, and operating context).

This special issue collects extended versions of four of the papers that have received the best scores during the review process of the 3rd IEEE International Conference on Autonomic Computing and Self- Organizing Systems (ACSOS 2022). In this introduction we are going to frame the papers in the general ACSOS context.

Once deployed in the real world, autonomous underwater vehicles (AUVs) are out of reach for human supervision yet need to take decisions to adapt to unstable and unpredictable environments. To facilitate research on self-adaptive AUVs, this paper presents SUAVE, an exemplar for two-layered system-level adaptation of AUVs, which clearly separates the application and self-adaptation concerns. The exemplar focuses on a mission for underwater pipeline inspection by a single AUV, implemented as a ROS 2-based system. This mission must be completed while simultaneously accounting for uncertainties such as thruster failures and unfavorable environmental conditions. The paper discusses how SUAVE can be used with different self-adaptation frameworks, illustrated by an experiment using the Metacontrol framework to compare AUV behavior with and without self-adaptation. The experiment shows that the use of Metacontrol to adapt the AUV during its mission improves its performance when measured by the overall time taken to complete the mission or the length of the inspected pipeline.

Computing systems form the backbone of many areas in our society, from manufacturing to traffic control, healthcare, and financial systems. When software plays a vital role in the design, construction, and operation, these systems are referred to as software-intensive systems. Self-adaptation equips a software-intensive system with a feedback loop that either automates tasks that otherwise need to be performed by human operators or deals with uncertain conditions. Such feedback loops have found their way to a variety of practical applications; typical examples are an elastic cloud to adapt computing resources and automated server management to respond quickly to business needs. To gain insight into the motivations for applying self-adaptation in practice, the problems solved using self-adaptation and how these problems are solved, and the difficulties and risks that industry faces in adopting self-adaptation, we performed a large-scale survey. We received 184 valid responses from practitioners spread over 21 countries. Based on the analysis of the survey data, we provide an empirically grounded overview the of state of the practice in the application of self-adaptation. From that, we derive insights for researchers to check their current research with industrial needs, and for practitioners to compare their current practice in applying self-adaptation. These insights also provide opportunities for applying self-adaptation in practice and pave the way for future industry-research collaborations.

Contemporary deployment environments are volatile, with conditions that are often hard to predict in advance, demanding solutions that are able to learn how best to design a system at runtime from a set of available alternatives. While the self-adaptive systems community has devoted significant attention to online learning, there is less research specifically directed towards learning for open-ended architectural adaptation - where individual components represent alternatives that can be added and removed dynamically. In this paper we present the Emergent Web Server (EWS), an architecture-based adaptive web server with 42 unique compositions of alternative components that present different utility when subjected to different workload patterns. This artefact allows the exploration of online learning techniques that are specifically able to consider the composition of logic that comprises a given system, and how each piece of logic contributes to overall utility. It also allows the user to add new components at runtime (and so produce new composition options), and to remove existing components; both are likely to occur in systems where developers (or automated code generators) deploy new code on a continuous basis and identify code which has never performed well. Our exemplar bundles together a fully-functional web server, a number of pre-packaged online learning approaches, and utilities to integrate, evaluate, and compare new online learning approaches.

Self-adaptation equips a software system with a feedback loop that automates tasks that otherwise need to be performed by operators. Such feedback loops have found their way to a variety of practical applications, one typical example is an elastic cloud. Yet, the state of the practice in self-adaptation is currently not clear. To get insights into the use of self-adaptation in practice, we are running a large-scale survey with industry. This paper reports preliminary results based on survey data that we obtained from 113 practitioners spread over 16 countries, 62 of them work with concrete self-adaptive systems. We highlight the main insights obtained so far: motivations for self-adaptation, concrete use cases, and difficulties encountered when applying self-adaptation in practice. We conclude the paper with outlining our plans for the remainder of the study.

Cyber-physical systems (CPS) and IoT systems are nowadays commonly designed as self-adaptive, endowing them with the ability to dynamically reconFigure to reflect their changing environment. This adaptation concerns also the security, as one of the most important properties of these systems. Though the state of the art on adaptivity in terms of security related to these systems can often deal well with fully anticipated situations in the environment, it becomes a challenge to deal with situations that are not or only partially anticipated. This uncertainty is however omnipresent in these systems due to humans in the loop, open-endedness and only partial understanding of the processes happening in the environment. In this paper, we partially address this challenge by featuring an approach for tackling access control in face of partially unanticipated situations. We base our solution on special kind of aspects that build on existing access control system and create a second level of adaptation that addresses the partially unanticipated situations by modifying access control rules. The approach is based on our previous work where we have analyzed and classified uncertainty in security and trust in such systems and have outlined the idea of access-control related situational patterns. The aspects that we present in this paper serve as means for application-specific specialization of the situational patterns. We showcase our approach on a simplified but real-life example in the domain of Industry 4.0 that comes from one of our industrial projects.

Background: With the rising popularity of Artificial Intelligence (AI), there is a growing need to build large and complex AI-based systems in a cost-effective and manageable way. Like with traditional software, Technical Debt (TD) will emerge naturally over time in these systems, therefore leading to challenges and risks if not managed appropriately. The influence of data science and the stochastic nature of AI-based systems may also lead to new types of TD or antipatterns, which are not yet fully understood by researchers and practitioners. Objective: The goal of our study is to provide a clear overview and characterization of the types of TD (both established and new ones) that appear in AI-based systems, as well as the antipatterns and related solutions that have been proposed. Method: Following the process of a systematic mapping study, 21 primary studies are identified and analyzed. Results: Our results show that (i) established TD types, variations of them, and four new TD types (data, model, configuration, and ethics debt) are present in AI-based systems, (ii) 72 antipatterns are discussed in the literature, the majority related to data and model deficiencies, and (iii) 46 solutions have been proposed, either to address specific TD types, antipatterns, or TD in general. Conclusions: Our results can support AI professionals with reasoning about and communicating aspects of TD present in their systems. Additionally, they can serve as a foundation for future research to further our understanding of TD in AI-based systems.

Test scenario generation for testing automated and autonomous driving systems requires knowledge about the recurring traffic cases, known as scenario types. The most common approach in industry is to have experts create lists of scenario types. This poses the risk both that certain types are overlooked; and that the mental model that underlies the manual process is inadequate. We propose to extract scenario types from real driving data by clustering recorded scenario instances, which are composed of timeseries. Existing works in the domain of traffic data either cannot cope with multivariate timeseries; are limited to one or two vehicles per scenario instance; or they use handcrafted features that are based on the mental model of the data scientist. The latter suffers from similar shortcomings as manual scenario type derivation. Our approach clusters scenario instances relying as little as possible on a mental model. As such, we consider the approach an important complement to manual scenario type derivation. It may yield scenario types overlooked by the experts, and it may provide a different segmentation of a whole set of scenarios instances into scenario types, thus overall increasing confidence in the handcrafted scenario types. We present the application of the approach to a real driving dataset.

Abstract Smart system applications (SSAs)—a heterogeneous landscape of applications of Internet of things, cyber-physical systems, and smart sensing systems—are composed of autonomous yet inherently cooperating components. An important problem in this area is how to hoist the cooperation of software components forming dynamic groups—ensembles—at the architectural level of an SSA. This is hard since ensembles can overlap, be nested, and be dynamically formed and dismantled based on several criteria. A related problem is how to combine component and ensemble specification with a well-established language supported on multiple platforms. To target these problems, we propose a specification and implementation language Trait-based COmponent Ensemble Language (TCOEL) based on Scala internal DSL, to describe both the architecture and formation of dynamic ensembles of components and their functional internals. To raise the level of expressivity, we introduce the concept of domain-specific extensions (traits) to the TCOEL core to reflect different paradigms’ concerns—such as movement in a 2D map, state-space modeling of physical processes, and statistical reasoning about uncertainty. This allows for configuring TCOEL for the needs of a specific SSA use case and, at the same time, facilitates reuse. To evaluate TCOEL, we show how it can be beneficially used in addressing the coordination of agents in a RoboCup Rescue Simulation application.

As software architecture practice relies more and more on runtime data to inform decisions in continuous experimentation and self-adaptation, it is increasingly important to consider the quality of the data used as input to the different decision-making and prediction algorithms. One issue in data-driven decisions is that real-life data coming from running systems can contain invalid or wrong values which can bias the result of data analysis. Data-driven decision-making should therefore comprise detection and handling of data anomalies as an integral part of the process. However, currently, anomaly detection is either absent in runtime decision-making approaches for continuous experimentation and self-adaptation or difficult to tailor to domain-specific needs. In this paper, we contribute by proposing a framework that simplifies the detection of data anomalies in timeseries-outputs of running systems. The framework is generic, since it can be employed in different domains, and tunable, since it uses expert user input in tailoring anomaly detection to the needs and assumptions of each domain. We evaluate the feasibility of the framework by successfully applying it to detecting anomalies in a real-life timeseries dataset from the traffic domain.

More and more software-intensive systems employ machine learning and runtime optimization to improve their functionality by providing advanced features (e. g. personal driving assistants or recommendation engines). Such systems incorporate a number of smart software functions (SSFs) which gradually learn and adapt to the users' preferences. A key property of SSFs is their ability to learn based on data resulting from the interaction with the user (implicit and explicit feedback)-which we call trainability. Newly developed and enhanced features in a SSF must be evaluated based on their effect on the trainability of the system. Despite recent approaches for continuous deployment of machine learning systems, trainability evaluation is not yet part of continuous integration and deployment (CID) pipelines. In this paper, we describe the different facets of trainability for the development of SSFs. We also present our approach for automated trainability evaluation within an automotive CID framework which proposes to use automated quality gates for the continuous evaluation of machine learning models. The results from our indicative evaluation based on real data from eight BMW cars highlight the importance of continuous and rigorous trainability evaluation in the development of SSFs.

Consumption of data streams generated from IoT devices during IoT application development is gaining prominence as the data insights are paramount for building high-impact applications. IoT mashup tools, i.e. tools that aim to reduce the development effort in the context of IoT via graphical flow-based programming, suffer from various architectural limitations which prevent the usage of data analytics as part of the application logic. Moreover, the approach of flow-based programming is not conducive for stream processing. We introduce our new mashup tool aFlux based on actor system with concurrent and asynchronous execution semantics to overcome the prevalent architectural limitations and support in-built user-configurable stream processing capabilities. Furthermore, parametrizing the control points of stream processing in the tool enables non-experts to use various stream processing styles and deal with the subtle nuances of stream processing effortlessly. We validate the effectiveness of parametrization in a real-time traffic use case.

The pervasiveness and growing complexity of software systems are challenging software engineering to design systems that can adapt their behavior to withstand unpredictable, uncertain, and continuously changing execution environments. Control theoretical adaptation mechanisms have received growing interest from the software engineering community in the last few years for their mathematical grounding, allowing formal guarantees on the behavior of the controlled systems. However, most of these mechanisms are tailored to specific applications and can hardly be generalized into broadly applicable software design and development processes. This article discusses a reference control design process, from goal identification to the verification and validation of the controlled system. A taxonomy of the main control strategies is introduced, analyzing their applicability to software adaptation for both functional and nonfunctional goals. A brief extract on how to deal with uncertainty complements the discussion. Finally, the article highlights a set of open challenges, both for the software engineering and the control theory research communities.

The dynamic nature of complex Cyber-Physical Systems puts extra requirements on their functionalities: they not only need to be dependable, but also able to adapt to changing situations in their environment. When developing such systems, however, it is often impossible to explicitly design for all potential situations up front and provide corresponding strategies. Situations that come out of this “envelope of adaptability” can lead to problems that end up by applying an emergency fail-safe strategy to avoid complete system failure. The existing approaches to self-adaptation cannot typically cope with such situations better—while they are adaptive (and can apply learning) in choosing a strategy, they still rely on a pre-defined set of strategies not flexible enough to deal with those situations adequately. To alleviate this problem, we propose the concept of meta-adaptation strategies, which extends the limits of adaptability of a system by constructing new strategies at runtime to reflect the changes in the environment. Though the approach is generally applicable to most approaches to self-adaptation, we demonstrate our approach on IRM-SA—a design method and associated runtime model for self-adaptive distributed systems based on component ensembles. We exemplify the meta-adaptation strategies concept by providing three concrete meta-adaptation strategies and show its feasibility on an emergency coordination case study.

The advent of Big Data has created a rich set of diverse languages and tools for data manipulation and analytics within the Hadoop ecosystem. Pig has a prominent role within this ecosystem as a scripting layer-a convenient way to create analytics jobs that are issued for batch processing in a Hadoop cluster. In order to leverage the benefits of graphical domain specific languages, namely intuitive visual design and inspection, we implemented a web-based graphical tool called QryGraph that complements Pig in various ways. First, it allows a user to create Pig queries in a graphical editor and check their syntax. Second, it provides an administrative interface for managing the execution and overall lifecycle of Pig queries. Finally, it will allow for debugging by running queries on test data sets and for creating user-defined query sub-graphs that can be reused across different Pig queries.

Cyber-physical systems (CPS) are systems of cooperating autonomous components which closely interact with and control the physical environment. Being distributed and typically based on periodic activities, CPS have to cope with the problem that data capturing a distributed state of the system and its environment are inherently inaccurate (they represent belief on the state). In particular, this poses a problem when dependability is being pursued. In this paper we address this issue by modeling belief at the architecture level. In particular, we enhance the architecture by models describing belief inaccuracy over time. We exploit these models to quantify at runtime the impact of belief staleness on its inaccuracy. We then use this quantification to drive architectural adaptation with the aim to increase dependability of the running CPS system.