In the fast-evolving field of artificial intelligence, Reinforcement Learning (RL) plays a crucial role in developing agents that can make decisions. As these systems become increasingly complex, the need for standardized and automated training methods becomes apparent. This paper presents a rule-based framework that integrates Large Language Models (LLMs) and heuristic-based code detectors to ensure compliance with best practices in RL training pipelines. We define a set of architectural rules that target best practices in important areas of RL-based architectures, such as checkpoints, hyperparameter tuning, and agent configuration. We validated our approach through a large-scale industrial case study and ten open-source projects. The results show that LLM-based detectors generally outperform heuristic-based detectors, especially when handling more complex code patterns. This approach effectively identifies best practices with high precision and recall, demonstrating its practical applicability.

Machine learning (ML) has emerged as a transformative subject, using various algorithms to help systems analyze data and make predictions. Deep Learning (DL) uses neural networks to address hard problems. Reinforcement Learning (RL) is a way to solve problems by making consecutive decisions. Understanding ML systems based only on the source code is often a challenging task, especially for inexperienced developers. In a controlled experiment involving one hundred fifty-eight participants, we assessed the understandability of ML-based systems and workflows through source code inspection compared to semi-formal representations in models and metrics. We hypothesize that ML system diagrams modeling details of ML workflows and practices like transfer learning and checkpoints can enhance the understandability of ML practices in system design comprehension tasks, assessed through task correctness . Additionally, providing these sources could lead to an increase in task duration , and we expect a significant correlation between correctness and duration . Our findings show that providing semi-formal ML system diagrams with the source code improves the effectiveness of the correctness for the DL relevant tasks. The control group had an average correctness of 0.7121, while the experimental group had a higher average correctness of 0.7759. On the other hand, participants who received only the system source code showed slightly better performance in the correctness task (average correctness 0.6808) within the RL relevant tasks compared to those who also received the semi-formal diagrams (average correctness of 0.6612). However, no significant difference was found in the duration task between the two. The control group, for the DL relevant tasks, took an average of 1571.62 s, whereas the experimental group took an average of 1763.85 s. For the RL relevant tasks, the control group had an average of 1883.80 s, while the experimental group 1925.46 s. However, semi-formal ML system diagrams can benefit specific scenarios.

In machine learning (ML) and machine learning operations (MLOps), automation serves as a fundamental pillar, streamlining the deployment of ML models and representing an architectural quality aspect. Support for automation is especially relevant when dealing with ML deployments characterised by the continuous delivery of ML models. Taking automation in MLOps systems as an example, we present novel metrics that offer reliable insights into support for this vital quality attribute, validated by ordinal regression analysis. Our method introduces novel, technology-agnostic metrics aligned with typical Architectural Design Decisions (ADDs) for automation in MLOps. Through systematic processes, we demonstrate the feasibility of our approach in evaluating automation-related ADDs and decision options. Our approach can itself be automated within continuous integration/continuous delivery pipelines. It can also be modified and extended to evaluate any relevant architectural quality aspects, thereby assisting in enhancing compliance with non-functional requirements and streamlining development, quality assurance and release cycles. • Introduces a semi-automated method for assessing MLOps system architecture qualities. • Models twenty-two MLOps architectures and develops a reusable metamodel. • Defines technology-agnostic metrics for evaluating automation in MLOps. • Employs a systematic process for sourcing and assessing various MLOps architectures. • Validates metrics through ordinal regression for reliable prediction models.

In the dynamic landscape of artificial intelligence and machine learning, Reinforcement Learning (RL) has emerged as a powerful paradigm for training intelligent agents in sequential decision-making. As RL architectures progress in complexity, the need for informed decision-making regarding training strategies and related consequences on the software architecture becomes increasingly intricate. This work addresses this challenge by presenting the outcomes of a qualitative, in-depth study focused on best practices and patterns within training strategies for RL architectures, as articulated by practitioners. Leveraging a model-based qualitative research method, we introduce a formal architecture decision model to bridge the gap between scientific insights and practical implementation. We aim to enhance the understanding of practitioners' approaches in RL architecture. The paper analyzes 33 knowledge sources to discern established industrial practices, patterns, relationships, and decision drivers. Based on this knowledge, we introduce a formal Architectural Design Decision (ADD) model, encapsulating 6 decisions, 29 decision options, and 19 decision drivers, providing robust decision-making support for this critical facet of RL-based software architectures.

Machine Learning Operations (MLOps) is the practice of streamlining and optimising the machine learning (ML) workflow, from development to deployment, using DevOps (software development and IT operations) principles and ML-specific activities. Architectural descriptions of MLOps systems often consist of informal textual descriptions and informal graphical system diagrams that vary considerably in consistency, quality, detail, and content. Such descriptions only sometimes follow standards or schemata and may be hard to understand. We aimed to investigate informal textual descriptions and informal graphical MLOps system architecture representations and compare them with semi-formal MLOps system diagrams for those systems. We report on a controlled experiment with sixty-three participants investigating the understandability of MLOps system architecture descriptions based on informal and semi-formal representations. The results indicate that the understandability (quantified by task correctness) of MLOps system descriptions is significantly greater using supplementary semi-formal MLOps system diagrams, that using semi-formal MLOps system diagrams does not significantly increase task duration (and thus hinder understanding), and that task correctness is only significantly correlated with task duration when semi-formal MLOps system diagrams are provided.

Deploying machine learning models to production is challenging, partially due to the misalignment between software engineering and machine learning disciplines but also due to potential practitioner knowledge gaps. To reduce this gap and guide decision-making, we conducted a qualitative investigation into the technical challenges faced by practitioners based on studying the grey literature and applying the Straussian Grounded Theory research method. We modelled current practices in machine learning, resulting in a UML-based architectural design decision model based on current practitioner understanding of the domain and a subset of the decision space and identified seven architectural design decisions, various relations between them, twenty-six decision options and forty-four decision drivers in thirty-five sources. Our results intend to help bridge the gap between science and practice, increase understanding of how practitioners approach deployment of their solutions, and support practitioners in their decision-making.