This article develops inverse reinforcement learning (IRL) control algorithms for nonlinear networked control systems (NCSs) to mimic trajectories of a target system governed by an unknown optimal cost function, despite the presence of random data dropouts and external disturbances. Data dropouts occur during: 1) reception of target trajectory data by the controller; 2) reception of state feedback data by the controller; and 3) reception of control input data by the actuator. By organically integrating $H_{\infty } $ control to account for disturbances and dropout-induced uncertainty, a model-based IRL algorithm is first developed. Building on this, a neural-network-based data-driven IRL algorithm is developed to infer the cost function and optimal control policy using available data while reducing dependence on system models. The proposed methods enable effective trajectory imitation under partial model knowledge, data dropouts, and disturbances, as demonstrated through simulation studies.

This paper proposes a data-driven model-free inverse reinforcement learning (IRL) algorithm tailored for solving an inverse$H_{\infty } $control problem. In the problem, both an expert and a learner engage in$H_{\infty } $control to reject disturbances and the learner’s objective is to imitate the expert’s behavior by reconstructing the expert’s performance function through IRL techniques. Introducing zero-sum game principles, we first formulate a model-based single-loop IRL policy iteration algorithm that includes three key steps: updating the policy, action, and performance function using a new correction formula and the standard inverse optimal control principles. Building upon the model-based approach, we propose a model-free single-loop off-policy IRL algorithm that eliminates the need for initial stabilizing policies and prior knowledge of the dynamics of expert and learner. Also, we provide rigorous proof of convergence, stability, and Nash optimality to guarantee the effectiveness and reliability of the proposed algorithms. Furthermore, we showcase the efficiency of our algorithm through simulations and experiments, highlighting its advantages compared to the existing methods.Note to Practitioners—Generally, the cost function for optimal tracking or imitation control is manually defined, which is a challenging task and may result in large tracking errors and slow tracking. In such cases, IRL is a powerful tool for reconstructing proper cost functions. Real-world systems, as demonstrated in practical cases, are frequently exposed to external disturbances and come with unknown models. Employing$H_{\infty } $control is an effective strategy to handle disturbances. However, applying model-free IRL to solve the inverse problem of$H_{\infty } $control for imitation remains an underexplored domain. This paper explores model-free inverse$H_{\infty } $control for imitating expert behaviors, specifically addressing the time-consuming nature of the existing IRL studies that employ a two-loop iteration structure. We propose an efficient single-loop IRL algorithm with a new framework to do this. It is data-driven and model-free, eliminating the need to find an initial stabilizing control policy, which is typically challenging. Additionally, it ensures convergence, stability, and optimality with provable guarantees.

This article proposes inverse reinforcement learning (IRL) algorithms for tracking control of linear networked control systems under random state dropouts during wireless transmission. The controlled system aims to track the optimal trajectory of a target system, despite the cost function governing the target's behaviors being unknown. The problem is complicated by random state dropouts occurring in two crucial scenarios: 1) the reception of the target's state and 2) feedback of the controlled system's states. Our approach enables the controlled system to infer the target's cost function and optimal control policy, thereby facilitating effective tracking. Specifically, we develop a model-based IRL algorithm that integrates the Smith predictor for state estimation. Then, we advance a state-dropout-aware inverse Q-learning algorithm that uses solely accessible system data, eliminating the need for system models. The theoretical validity of the proposed algorithms is rigorously established, and their practical effectiveness is validated through numerical simulations.

This article studies the trajectory imitation control problem of linear systems suffering external disturbances and develops a data-driven static output feedback (OPFB) control-based inverse reinforcement learning (RL) approach. An Expert-Learner structure is considered where the learner aims to imitate expert's trajectory. Using only measured expert's and learner's own input and output data, the learner computes the policy of the expert by reconstructing its unknown value function weights and thus, imitates its optimally operating trajectory. Three static OPFB inverse RL algorithms are proposed. The first algorithm is a model-based scheme and serves as basis. The second algorithm is a data-driven method using input-state data. The third algorithm is a data-driven method using only input-output data. The stability, convergence, optimality, and robustness are well analyzed. Finally, simulation experiments are conducted to verify the proposed algorithms.

Intelligent manufacturing has promoted the development of Industry 4.0 and enabled the manufacturing industry to gradually move into the stage of intelligence with the rapid development of the Internet of Things and the Industrial Internet. An intelligent manufacturing system is a manufacturing system that can automatically adapt to changing environments and varying process requirements with minimal supervision and assistance from operators. Therefore, intelligent manufacturing has become a recognized core high technology to enhance the overall competitiveness of the manufacturing industry. The goal of intelligent manufacturing is to make production resources form a circular network with the characteristics of autonomy, adjustability, and configurability, to develop production processes flexibly, and to realize the efficiency of individual customization. For example, by analyzing the factory floor data, equipment monitored data, and the enterprise manufacturing database, it could help to store, explore, and make complex decisions for the manufacturing system. To achieve this goal, modern information technologies, such as artificial intelligence, big data, cloud computing, and mobile Internet, modeling, control, and optimization need to be integrated and collaborated with the physical resources of the manufacturing process, which triggers new theory, solution algorithms, and application scenarios.

In inverse reinforcement learning (RL), there are two agents. An expert target agent has a performance cost function and exhibits control and state behaviors to a learner. The learner agent does not know the expert's performance cost function but seeks to reconstruct it by observing the expert's behaviors and tries to imitate these behaviors optimally by its own response. In this article, we formulate an imitation problem where the optimal performance intent of a discrete-time (DT) expert target agent is unknown to a DT Learner agent. Using only the observed expert's behavior trajectory, the learner seeks to determine a cost function that yields the same optimal feedback gain as the expert's, and thus, imitates the optimal response of the expert. We develop an inverse RL approach with a new scheme to solve the behavior imitation problem. The approach consists of a cost function update based on an extension of RL policy iteration and inverse optimal control, and a control policy update based on optimal control. Then, under this scheme, we develop an inverse reinforcement Q-learning algorithm, which is an extension of RL Q-learning. This algorithm does not require any knowledge of agent dynamics. Proofs of stability, convergence, and optimality are given. A key property about the nonunique solution is also shown. Finally, simulation experiments are presented to show the effectiveness of the new approach.

static OPFB control problem of linear discrete-time (DT) systems. The primary contribution of the proposed algorithms lies in a newly developed OPFB control algorithm form for completely unknown systems. Under the premise of satisfying disturbance attenuation conditions, the conditions for the existence of the optimal OPFB solution are given. The convergence of the proposed Q -learning methods, and the difference and equivalence of two algorithms are rigorously proven. Moreover, considering the effects brought by probing noise for the persistence of excitation (PE), the proposed off-policy Q -learning method has the advantage of being immune to probing noise and avoiding biasedness of solution. Simulation results are presented to verify the effectiveness of the proposed approaches.

This article provides a novel inverse reinforcement learning (RL) algorithm that learns an unknown performance objective function for tracking control. The algorithm combines three steps: 1) an optimal control update; 2) a gradient descent correction step; and 3) an inverse optimal control (IOC) update. The new algorithm clarifies the relation between inverse RL and IOC. It is shown that the reward weight of an unknown performance objective that generates a target control policy may not be unique. We characterize the set of all weights that generate the same target control policy. We develop a model-based algorithm and, further, two model-free algorithms for systems with unknown model information. Finally, simulation experiments are presented to show the effectiveness of the proposed algorithms.

This article applies a singular perturbation theory to solve an optimal linear quadratic tracker problem for a continuous-time two-time-scale process. Previously, singular perturbation was applied for system regulation. It is shown that the two-time-scale tracking problem can be separated into a linear-quadratic tracker (LQT) problem for the slow system and a linear-quadratic regulator (LQR) problem for the fast system. We prove that the solutions to these two reduced-order control problems can approximate the LQT solution of the original control problem. The reduced-order slow LQT and fast LQR control problems are solved by off-policy integral reinforcement learning (IRL) using only measured data from the system. To test the effectiveness of the proposed method, we use an industrial thickening process as a simulation example and compare our method to a method with the known system model and a method without time-scale separation.

This paper presents a model-free optimal approach based on reinforcement learning for solving the output regulation problem for discrete-time systems under disturbances. This problem is first broken down into two optimization problems: 1) a constrained static optimization problem is established to find the solution to the output regulator equations (i.e., the feedforward control input) and 2) a dynamic optimization problem is established to find the optimal feedback control input. Solving these optimization problems requires the knowledge of the system dynamics. To obviate this requirement, a model-free off-policy algorithm is presented to find the solution to the dynamic optimization problem using only measured data. Then, based on the solution to the dynamic optimization problem, a model-free approach is provided for the static optimization problem. It is shown that the proposed algorithm is insensitive to the probing noise added to the control input for satisfying the persistence of excitation condition. Simulation results are provided to verify the effectiveness of the proposed approach.

Current challenges in industrial processes control include achieving optimum operation for systems with two-time-scale dynamics and unknown models. This paper presents, for the first time, the integration of singular perturbation theory and reinforcement learning to solve this problem. To this end, an optimal operational control (OOC) problem with two time scales is formulated to reach the desired operational indices. Then, a singularly perturbed dynamics for two-time-scale industrial operational processes is developed by introducing a perturbed scale, resulting in the separation of the original system dynamics. Thus, the original optimization problem is decomposed into a reduced slow subproblem and a boundary fast subproblem. The fact that the sum of the separate solutions of these subproblems is approximately equal to the solution of the OOC problem is proven. Then, two Q-learning algorithms are proposed to obtain a composite feedback control. Finally, an industrial thickener example is employed to show the effectiveness of the proposed method.

In this article, a new model-free approach is proposed to solve the output regulation problem for networked control systems, where the system state can be lost in the feedback process. The goal of the output regulation is to design a control law that can make the system achieve asymptotic stability of the tracking error while maintaining the stability of the closed-loop system. The solvability of the output regulation problem depends on the solvability of a set of matrix equations called the regulator equations. First, a restructured dynamic system is established by using the Smith predictor; then, an off-policy algorithm based on reinforcement learning is developed to calculate the feedback gain using only the measured data when dropout occurs. Based on the solution to the feedback gain, a model-free solution is provided for solving the forward gain using the regulator equations. The simulation results demonstrate the effectiveness of the proposed approach for discrete-time networked systems with unknown dynamics and dropout.

This paper studies the operational optimal control problem for the industrial flotation process, a key component in the mineral processing concentrator line. A new model-free data-driven method is developed here for real-time solution of this problem. A novel formulation is given for the optimal selection of the process control inputs that guarantees optimal tracking of the operational indices while maintaining the inputs within specified bounds. Proper tracking of prescribed operational indices, namely concentrate grade and tail grade, is essential in the proper economic operation of the flotation process. The difficulty in establishing an accurate mathematic model is overcome, and optimal controls are learned online in real time, using a novel form of reinforcement learning we call interleaved learning for online computation of the operational optimal control solution. Simulation experiments are provided to verify the effectiveness of the proposed interleaved learning method and to show that it performs significantly better than standard policy iteration and value iteration.

This paper develops a new method for solving the optimal control tracking problem for networked control systems (NCSs), where network-induced dropout can occur and the system dynamics are unknown. First, a novel dropout Smith predictor is designed to predict the current state based on historical data measurements over the communication network. Then, it is shown that the quadratic form of the performance index is preserved even with dropout, and the optimal tracker solution with dropout is given based on a novel dropout generalized algebraic Riccati equation. New algorithms for off-line policy iteration (PI), online PI, and Q-learning PI are presented for NCS with dropout. The Q-learning algorithm adaptively learns the optimal control online using data measured over the communication network based on reinforcement learning, including dropout, without requiring any knowledge of the system dynamics. Simulation results are provided to show that the proposed approaches give proper optimal tracking performance for the NCS with unknown dynamics and dropout.

Industrial flow lines are composed of unit processes operating on a fast time scale and performance measurements known as operational indices measured at a slower time scale. This paper presents a model-free optimal solution to a class of two time-scale industrial processes using off-policy reinforcement learning (RL). First, the lower-layer unit process control loop with a fast sampling period and the upper-layer operational index dynamics at a slow time scale are modeled. Second, a general optimal operational control problem is formulated to optimally prescribe the set-points for the unit industrial process. Then, a zero-sum game off-policy RL algorithm is developed to find the optimal set-points by using data measured in real-time. Finally, a simulation experiment is employed for an industrial flotation process to show the effectiveness of the proposed method.

In this paper, the performance-based control design problem for double-layer networked industrial processes is investigated. At the device layer, the prescribed performance functions are first given to describe the output tracking performance, and then by using backstepping technique, new adaptive fuzzy controllers are designed to guarantee the tracking performance under the effects of input dead-zone and the constraint of prescribed tracking performance functions. At operation layer, by considering the stochastic disturbance, actual index value, target index value, and index prediction simultaneously, an adaptive inverse optimal controller in discrete-time form is designed to optimize the overall performance and stabilize the overall nonlinear system. Finally, a simulation example of continuous stirred tank reactor system is presented to show the effectiveness of the proposed control method.

Considering an optimal investment problem for a retailer in electricity market, the objective is to seek the optimal investment decision that maximizes the weighted sum of the expected return and the variance of wealth. Unlike existing works, the price fluctuation of both the wholesale and retail side of electricity market is considered, and the retailer can invest its wealth in electricity market and traditional financial market simultaneously. Hence, there is a complicated wealth dynamic, which is the main challenge in our work. In this paper, by utilizing the method of Lagrange multiplier and the classical Tchebycheff inequality, we first show that the investment problem is a quadratic programming problem in terms of the decision variable, and thus has a unique optimal solution. Then, a closed-form optimal solution is derived by solving the stationary equation and comparing the feasible solution interval. Based on the optimal solution, we find the key price, which will affect the investment is the wholesale price rather than the retail price. Moreover, with a similar analysis approach, we also provide the optimal solution considering a more general model, which allows the retailer to purchase the electricity temporarily to avoid the supply shortage. Extensive simulations demonstrate the better performance of the proposed solution over the Kelly strategy widely used in the financial market.

Recently, content-based publish/subscribe (pub/sub) services have become a significant research field in opportunistic mobile networks (OppNets). Pub/sub is an asynchronous messaging paradigm, in which content transmissions are guided by the interest. Since selfish behavior is common in reality, nodes often behave selfishly with an aim to maximize their own utilities without considering performance of other nodes. Therefore, how to encourage nodes to collect, store and share network content efficiently is one of the key challenges under this paradigm. In this paper, we propose an incentive-based pub/sub scheme, called ConSub, for OppNets. In ConSub, Tit-For-Tat (TFT) mechanism is employed to deal with selfish behavior. ConSub also implements a content exchange protocol between two interacting node, thus encouraging them to play as businessmen and carry contents to satisfy each other's interest. Specifically, the exchange order is determined by the content utility, which is calculated by contact probability and cooperation level between the current node and its neighbors subscribing to the interest. Extensive realistic trace-driven simulation results show that ConSub is superior to existing schemes in terms of delivered packets and transmission hops with reasonable transmission cost.

Complex industrial processes are controlled by the local regulation controllers at the field level, and the setpoints for the regulation are usually made by manual decomposition of the overall economic objective according to the operators' experience. If a precise static process model can be built, real-time optimization (RTO) can be used to generate the setpoints. Nevertheless, since the aforementioned control structure is actually open-loop, the desired economic objective of the whole processes may not be tracked when disturbances exist. Aiming at solving this problem, a novel network based model predictive control method (MPC) for setpoints compensation is proposed in this paper. Firstly, a multivariable proportional integral (PI) controller is designed to perform the local regulation control. Secondly, a stochastic packet dropout model is adopted to characterize the measurement and human-in-the-loop delay effect. Then, a model predictive controller considering the random dropout effect is developed to compensate the setpoints dynamically according to the changing conditions of the processes, such that the prescribed performance objective can be obtained. Finally, a flotation process model is employed to demonstrate the effectiveness of the proposed method.

In this paper, we consider the issue of data broadcasting in mobile social networks (MSNets). The objective is to broadcast data from a superuser to other users in the network. There are two main challenges under this paradigm, namely 1) how to represent and characterize user mobility in realistic MSNets; 2) given the knowledge of regular users' movements, how to design an efficient superuser route to broadcast data actively. We first explore several realistic data sets to reveal both geographic and social regularities of human mobility, and further propose the concepts of geocommunity and geocentrality into MSNet analysis. Then, we employ a semi-Markov process to model user mobility based on the geocommunity structure of the network. Correspondingly, the geocentrality indicating the “dynamic user density” of each geocommunity can be derived from the semi-Markov model. Finally, considering the geocentrality information, we provide different route algorithms to cater to the superuser that wants to either minimize total duration or maximize dissemination ratio. To the best of our knowledge, this work is the first to study data broadcasting in a realistic MSNet setting. Extensive trace-driven simulations show that our approach consistently outperforms other existing superuser route design algorithms in terms of dissemination ratio and energy efficiency.

Mobility pattern of device users plays a crucial role in a wide range of mobile computing applications, including data forwarding, content sharing, information search and advertising. Hence, it is important to characterize the mobility path information of users, so as to accurately predict user mobility. In this paper, we introduce two typical user mobility patterns: standard Markov and semi-Markov models. Especially, we experimentally explore the correlation of community and geography information in Mobile Social Networks (MSNets), and analyze user sojourn time distribution over communities. Both of theoretical analysis and trace-driven simulation results show that semi-Markov model is more effective in characterizing user mobility pattern and further making more accurate mobility prediction compared with standard Markov model.

In mobile social networks (MSNets), data dissemination is an important topic, which has not been widely investigated yet. Active data dissemination is a networking paradigm where a superuser intentionally facilitates the connectivity in the network. One of the key challenges under this paradigm is how to design the most efficient superuser route to achieve certain properties of end-to-end connectivity. Most existing solutions only focus on the network with stationary users or strongly constrained node mobility, and assume the superuser always moves with a fixed route. In this paper, we propose a flexible approach to design the superuser routes, considering the realistic user movements in MSNets. To the best of our knowledge, this work is the first to study active data dissemination from the social network perspective. We explore the geographic regularity of human mobility in the network, employ a semi-Markov analytical model to describe such mobility pattern, and hence formulate the superuser route design as a combinational optimization problem of Convex Optimization and Traveling Salesman Problem by exploiting social network concepts including communities and centrality. Extensive trace-driven simulations show that our approach consistently outperforms other existing superuser route design algorithms in terms of delivery ratio and energy efficiency.

Protocol testing has been one of the most active fields in computer networks. In Wireless Sensor Networks (WSNs), before directly employing protocols on hardware testbed, we expect an effective test framework to verify the logical correctness of protocols on computers, which could facilitate the whole test process. Towards this goal, this paper introduces an upper test framework in WSNs, namely Protocol Testing Framework for WSNs (PTFW).