This article introduces a novel integration of deep model-based learning with modular state-based Stackelberg games (Mod-SbSG) for distributed self-optimization in manufacturing systems, using a sample-efficient approach. Model-free Mod-SbSG requires frequent interactions with real systems to find optimal solutions, which can be costly, time-consuming, and risky in industrial settings. Prior studies handled this by using digital representations to train Mod-SbSG players, but accurate representations are often difficult to develop. Hence, our framework replaces digital representations with deep learning methods that learn system dynamics, optimize policies within Mod-SbSG, and reduce real-world interactions. The method includes two main steps: 1) designing deep learning models to predict system dynamics and 2) training Mod-SbSG players in virtual environments. We evaluate single- and multistep predictors and demonstrate network reuse for transfer learning in adaptable systems, which reduces real system interactions by 77.78% in a laboratory testbed industrial control scenario.

This paper presents the novel RLPID architecture for online-adaptive control, which combines classical proportional-integral-derivative (PID) control with reinforcement learning (RL). This hybrid approach enables dynamic online adjustment of PID parameters during control operation. Specifically, we propose a multi-objective reward structure that integrates established control criteria and analyze suitable configurations for different system dynamics. The RLPID controller has been implemented within the open-source middleware MLPro, where it is embedded in newly developed sub-frameworks for classical and online-adaptive control. Owing to its hybrid nature, the architecture can be used both in traditional control loops and within the Markov decision process of RL. Its effectiveness and practical applicability are demonstrated in a cascade control scenario.

This article presents a novel game-theoretical (GT) framework, distributed Stackelberg strategies in state-based potential games (DS2-SbPGs), for autonomous multiobjective optimization in decentralized manufacturing systems. Existing approaches, including multiagent reinforcement learning (MARL) and native SbPG, struggle with scalability, coordination inefficiencies, and the complexity of tuning combined objective functions in real-world settings. DS2-SbPG integrates potential games and Stackelberg games, which improves the cooperative tradeoff capabilities of potential games and the multiobjective optimization handling by Stackelberg games. Notably, all training procedures are conducted in a fully distributed manner. DS2-SbPG offers a promising solution to finding optimal tradeoffs between objectives by eliminating the complexities of setting up combined objective optimization functions for individual players in self-learning domains, particularly in real-world industrial settings with diverse and numerous objectives between the subsystems. We formally prove that DS2-SbPG constitutes a dynamic potential game with guaranteed convergence. Experimental validation on a laboratory-scale testbed demonstrates the effectiveness of DS2-SbPG and its two variants: one with a single-leader–follower structure and another (Stack DS2-SbPG) for multileader–follower scenarios. Both variants significantly outperform native SbPG, achieving up to 10.61% reduction in power consumption while enhancing overall system performance, which signals the potential of DS2-SbPG in real-world applications.

In this work we propose a novel approach based on model-based Reinforcement Learning (RL) for the sim-to-real transfer of industrial robots. Specifically, we propose to employ a recently developed kinematics-informed, modular neural network serving as a learnable environment model within the world model framework. Using the kinematics-informed model, training of the world model is made more efficient resulting in faster training. Furthermore, the approach allows to train industrial robots on specific tasks solely within the simulation of the system thereby saving time and energy-consumption. Using simulations ensures safe and controlled training implementation and allows for parallelization to increase training speed. We conduct various experiments which underline the effectiveness of the proposed method. We show that training the RL algorithm solely within the simulation, results in a hundred percent task completion rate in both simulation and real world experiments.

Tiny ML, defined by its ability to integrate AI into the smallest devices, unlocks the potential for always-on AI solutions at the edge. This capability is essential in predictive maintenance, enabling smart sensors within the Industrial In-ternet of Things (1IoT). In factory automation, the ability to detect anomalies and adapt those anomaly detection models to varying operating conditions is crucial for the efficiency and longevity of machinery. However, the challenge of semi-supervised anomaly detection with domain adaptation across diverse working conditions has not been sufficiently addressed in TinyML. This paper presents Shared Encoder Domain Adaptation (SEDA) to overcome these limitations. SEDA is a multi-target domain adaptation method for anomaly detection tailored to TinyML applications. Our approach facilitates the transfer of knowledge from existing domains to new ones, ensuring reliable anomaly detection in all domains with minimal data under new operational conditions. This is particularly relevant for IloT applications, where sensors must perform condition monitoring in a self-learning manner. The effectiveness of our method is evaluated through an extensive parameter study on the Toy-ADMOS 2 data set and compared to various baseline methods. The results show an improvement of up to 0.212 on the Area Under the Receiver Operating Characteristic Curve (ROC-AUC) score. It demonstrates that our method significantly enhances performance in multi-target domain adaptation. Importantly, it does this while remaining efficient for resource-constrained systems. This method has the potential to enable self-learning sensor systems with adaptation strategies to different working conditions while requiring only a few data points from these new domains.

In this paper, we introduce novel gradient-based optimization methods for state-based potential games (SbPGs) within self-learning distributed production systems. SbPGs are recognised for their efficacy in enabling self-optimizing distributed multi-agent systems and offer a proven convergence guarantee, which facilitates collaborative player efforts towards global objectives. Our study strives to replace conventional ad-hoc random exploration-based learning in SbPGs with contemporary gradient-based approaches, which aim for faster convergence and smoother exploration dynamics, thereby shortening training duration while upholding the efficacy of SbPGs. Moreover, we propose three distinct variants for estimating the objective function of gradient-based learning, each developed to suit the unique characteristics of the systems under consideration. To validate our methodology, we apply it to a laboratory testbed, namely Bulk Good Laboratory Plant, which represents a smart and flexible distributed multi-agent production system. The incorporation of gradient-based learning in SbPGs reduces training times and achieves more optimal policies than its baseline.

This article describes a novel concept to optimize manufacturing systems distributively through data-based learning. We propose a game-theoretic (GT) learning set-up that is incorporated with accessible control code of the programmable logic controller (PLC) to accelerate the optimal policies learning procedures, instead of learning everything from scratch. Therefore, we offer to process the accessible and available control code into a GT-based learning framework which is subsequently optimized in a fully distributed manner. To this end, we employ the recently developed framework of state-based potential games (PGs) and prove that under mild conditions PLC-informed (PLCi) learning forms a state-based PG framework. We conduct the experiment on a laboratory scale testbed in numerous production scenarios. The experiment's results highlight the major potential of using the PLCi GT-learning, which is the reduction of energy consumption of the production timescales and improvement of production efficiency while nearly halven the learning times.

In this paper, we discuss the impact of communication and memory-based learners on distributed self-optimization of smart and flexible manufacturing units. Specifically, we employ the recently proposed framework of state-based potential games, which has proven to be successful in allowing distributed optimization in multi-agent systems. We first augment the framework with additional communication capabilities for the individual players and analyze the efficacy of state and action communications within the different players. Second, we incorporate memory states within the learning dynamics of the players and analyze their impact on the learning performance. The proposed method is inspired by the promising results of memory-based reinforcement learning. However, previous studies have rarely dealt with distributed manufacturing control. We believe that it will be important to explore the potential use of the communication and memory-based approaches in manufacturing control with multi-agent settings. Hence, the proposed method is applied to a bulk good laboratory plant providing a thorough experimental analysis of the effect of the various improvements with very encouraging results.

This article presents a novel approach for distributed optimization of production units based on potential game (PG) theory and machine learning. The core of our approach is split into two parts: the first part concentrates on the conceptual treatment of modular installed production units in terms of a PG scenario. The second part focuses on the development and incorporation of suitable learning algorithms to finally form an intelligent autonomous system. In this context, we model the production environment as a state-based PG where each actuator of each module has the role of an agent in the game aiming to maximize its utility value by learning the optimal process behavior. The benefit of the additional state information is visible in the performance of the algorithm making the environment dynamic and serving as a connector between the players. We propose a novel learning algorithm based on a global interpolation method that is applied to a laboratory scale modular bulk good system. The thorough analysis of the encouraging results yields to highly interesting insights into the learning dynamics and the process itself. The benefits of our distributed optimization approach are the plug-and-play functionality, the online capability, fast adaption to changing production requirements, and the possibility of an IEC 61131 conforming to PLC implementation.

We present a novel approach for distributed optimization of highly flexible, modular production units enabling plug-and-play production with online optimization capabilities to adapt fast to changing production requirements. The approach is fully distributed in the sense that each production module to be optimized is equipped with its own optimization agent which local objective is the optimization of its own production objectives. To assure the necessary coordination between the agents, the resulting distributed optimization problem is designed using concepts of game theory. To this end, we model the production environment in terms of a potential game where each module is modeled as a player of the game. By assigning suitable utility functions to the players coordination of the agents behavior is achieved to find an optimal collective behavior. We apply the approach to a laboratory scale distributed bulk good production testbed with very encouraging results. In addition, due to the computational simplicity of the approach, an implementation in IEC61131 compatible code is possible allowing a direct implementation of the approach in existing production units.

This paper introduces a novel approach for the control of flexible manufacturing units by means of cooperatively interacting industrial robots. For fast adoption to the actual production requirements, we embed a learning module into the manufacturing cell. This learning module allows the robots to learn to solve the given task with respect to a given optimization objective. Simultaneously, the robots learn to efficiently cooperate and find an optimal collective behavior while solving the task. To this end, we develop two different control algorithms based on reinforcement learning. The first approach is based on a centralized agent which coordinates the learning behavior of the whole manufacturing cell. In the second approach, a learning agent is assigned to each robot allowing for more flexibility and reducing the state-action space of the reinforcement learning problem at hand. The approaches are applied to a laboratory testbed using two cooperating industrial robots which should learn to optimize the throughput of the manufacturing cell. A comparison of both approaches shows the improved performance of the decentralized learning agents compared to the centralized one both in goal achievement and learning speed.

This paper introduces a novel approach for self-learning in highly flexible, modular manufacturing systems enabling fast reconfiguration and online adaptation to changing production requirements. The approach is based on a distributed optimization scheme such that production modules are equipped with their own optimization agent with its local objectives to be optimized. The communication and coordination of the agent is limited to the basically required amount. The approach is based on the recently developed deep deterministic policy gradient (DDPG) approach, a high performing algorithm from the family of actor-critic reinforcement learning algorithms. As DDPG is based on single agent learning, we develop a fully distributed multi-agent learning setting with different levels of information about the neighbors. We apply the approach to a laboratory scale distributed bulk good production testbed with very encouraging results. Particularly, we found very reasonable control strategies by learning the agents from scratch.

This paper presents a novel approach to operate industrial robots as used for manufacturing lines within a cooperative robot station. The proposed framework consists of the application of especially to the cooperative robot handling problem adjusted Reinforcement Learning (RL) algorithms. Such RL-algorithms deal with sequential decision making processes in a trial-and-error learning interaction with the environment, to finally gain an optimal team-working behavior among the robots. In particular application results to a real team-working robot station underline the effectiveness of the novel RL approach.

This paper presents a novel approach for self-reconfiguration and plug-and-play control of modular production units. The approach is based on a service-oriented architecture which allows for a fully decentralized control and communication structure. Each system module is equipped with its own control system and communicates with other units by means of predefined service requests. Hence, the control of the overall system is solely executed by the modules own control and by the exchange of service requests of each station. The approach is implemented on a laboratory bulk good system typically used in the pharmaceutical industry with modular system structure. Results obtained from experiments underline the applicability of the approach.

This paper presents a novel approach to the optimization of energy consumption in large scale industrial bulk good processes. The approach is based on a model-free self-learning algorithm solely based on available process data using ideas from the well known reinforcement learning framework. To this end energy consumers of the plant are integrated in the optimization framework such that each consumer learns its own optimal energy profile for a given production task. The approach is implemented on a laboratory size testbed where the task is the supply of bulk good to a subsequent dosing section. The capability of the approach is underlined by the results obtained at the testbed.