This article develops inverse reinforcement learning (IRL) control algorithms for nonlinear networked control systems (NCSs) to mimic trajectories of a target system governed by an unknown optimal cost function, despite the presence of random data dropouts and external disturbances. Data dropouts occur during: 1) reception of target trajectory data by the controller; 2) reception of state feedback data by the controller; and 3) reception of control input data by the actuator. By organically integrating $H_{\infty } $ control to account for disturbances and dropout-induced uncertainty, a model-based IRL algorithm is first developed. Building on this, a neural-network-based data-driven IRL algorithm is developed to infer the cost function and optimal control policy using available data while reducing dependence on system models. The proposed methods enable effective trajectory imitation under partial model knowledge, data dropouts, and disturbances, as demonstrated through simulation studies.

This article addresses the existing reinforcement learning (RL) control issues of continuous-time Markov jump systems (MJSs), including their synchronous iteration structure using nonlatest updates, nonmonotonic convergence, and requirement of initial admissible control and all-time persistent excitation (PE) condition. We propose advanced model-based and model-free RL algorithms that: 1) have asynchronous decoupled Lyapunov iteration equations to approximate the optimal control solutions using the latest updates; 2) determine the initial admissible control policy (IACP) and initial value function matrix without relying on engineering experiences; and 3) relax PE condition with a milder initial excitation (IE) condition. Rigorous theoretical analyses are provided to establish the monotonic convergence to the optimal control solution and the closed-loop stability at each iteration. Finally, the simulation and comparison results verify the proposed algorithms.

Deep predictive coding networks (DPCNs) effectively model and capture video features through a bi-directional inference without labels. They are based on an overcomplete description of video scenes, and one of the bottlenecks has been the lack of effective sparsification techniques to find discriminative and robust dictionaries. This paper proposes a DPCN with a fast inference of internal dictionaries and variables that achieve high sparsity and feature clustering accuracy. The proposed unsupervised learning procedure uses majorization-minimization (MM) to smooth sparsity constraints in optimization and admits explainability and convergence. Experiments in the image and video data sets CIFAR-10, Super Mario Bros, and Coil-100 validate that the approach outperforms previous versions of DPCNs on learning rate, sparsity ratio, and feature clustering accuracy. This advance opens the door for general applications in object recognition in video without labels.

This paper proposes a data-driven model-free inverse reinforcement learning (IRL) algorithm tailored for solving an inverse$H_{\infty } $control problem. In the problem, both an expert and a learner engage in$H_{\infty } $control to reject disturbances and the learner’s objective is to imitate the expert’s behavior by reconstructing the expert’s performance function through IRL techniques. Introducing zero-sum game principles, we first formulate a model-based single-loop IRL policy iteration algorithm that includes three key steps: updating the policy, action, and performance function using a new correction formula and the standard inverse optimal control principles. Building upon the model-based approach, we propose a model-free single-loop off-policy IRL algorithm that eliminates the need for initial stabilizing policies and prior knowledge of the dynamics of expert and learner. Also, we provide rigorous proof of convergence, stability, and Nash optimality to guarantee the effectiveness and reliability of the proposed algorithms. Furthermore, we showcase the efficiency of our algorithm through simulations and experiments, highlighting its advantages compared to the existing methods.Note to Practitioners—Generally, the cost function for optimal tracking or imitation control is manually defined, which is a challenging task and may result in large tracking errors and slow tracking. In such cases, IRL is a powerful tool for reconstructing proper cost functions. Real-world systems, as demonstrated in practical cases, are frequently exposed to external disturbances and come with unknown models. Employing$H_{\infty } $control is an effective strategy to handle disturbances. However, applying model-free IRL to solve the inverse problem of$H_{\infty } $control for imitation remains an underexplored domain. This paper explores model-free inverse$H_{\infty } $control for imitating expert behaviors, specifically addressing the time-consuming nature of the existing IRL studies that employ a two-loop iteration structure. We propose an efficient single-loop IRL algorithm with a new framework to do this. It is data-driven and model-free, eliminating the need to find an initial stabilizing control policy, which is typically challenging. Additionally, it ensures convergence, stability, and optimality with provable guarantees.

This paper studies multiplayer multiagent differential graphical games, where each agent represents a linear dynamic system with multiple heterogeneous control inputs, referred to as players. The objective of the games is to control all followers to achieve containment within the convex hull of the leaders. Meanwhile, each player minimizes its cost function, which is also jointly affected by the other players and neighboring states in the same and neighboring agents. We design global Nash equilibrium (NE) (called Nash) control policies in the games, enabling all players to play the optimal control policies. The global Nash control solution is ensured to be distributed as it is computed using only local agents’ states. The solvability of the games, the asymptotic stability of the local error system, and the distributed global NE property are guaranteed. In addition, a data-driven integral reinforcement learning (RL) algorithm ensures the online computation of the distributed Nash control without requiring explicit knowledge of the system model matrices, by utilizing online measurements of system trajectories. The algorithm solves optimal control and inverse optimal control (IOC) as subproblems. A simulation example validates the algorithms.

This article proposes inverse reinforcement learning (IRL) algorithms for tracking control of linear networked control systems under random state dropouts during wireless transmission. The controlled system aims to track the optimal trajectory of a target system, despite the cost function governing the target's behaviors being unknown. The problem is complicated by random state dropouts occurring in two crucial scenarios: 1) the reception of the target's state and 2) feedback of the controlled system's states. Our approach enables the controlled system to infer the target's cost function and optimal control policy, thereby facilitating effective tracking. Specifically, we develop a model-based IRL algorithm that integrates the Smith predictor for state estimation. Then, we advance a state-dropout-aware inverse Q-learning algorithm that uses solely accessible system data, eliminating the need for system models. The theoretical validity of the proposed algorithms is rigorously established, and their practical effectiveness is validated through numerical simulations.

This article proposes a data-driven model-free inverse Q-learning algorithm for continuous-time linear quadratic regulators (LQRs). Using an agent's trajectories of states and optimal control inputs, the algorithm reconstructs its cost function that captures the same trajectories. This article first poses a model-based inverse value iteration scheme using the agent's system dynamics. Then, an online model-free inverse Q-learning algorithm is developed to recover the agent's cost function only using the demonstrated trajectories. It is more efficient than the existing inverse reinforcement learning (RL) algorithms as it avoids the repetitive RL in inner loops. The proposed algorithms do not need initial stabilizing control policies and solve for unbiased solutions. The proposed algorithm's asymptotic stability, convergence, and robustness are guaranteed. Theoretical analysis and simulation examples show the effectiveness and advantages of the proposed algorithms.

This article addresses the problem of learning the objective function of linear discrete-time systems that use static output-feedback (OPFB) control by designing inverse reinforcement learning (RL) algorithms. Most of the existing inverse RL methods require the availability of states and state-feedback control from the expert or demonstrated system. In contrast, this article considers inverse RL in a more general case where the demonstrated system uses static OPFB control with only input-output measurements available. We first develop a model-based inverse RL algorithm to reconstruct an input-output objective function of a demonstrated discrete-time system using its system dynamics and the OPFB gain. This objective function infers the demonstrations and OPFB gain of the demonstrated system. Then, an input-output Q -function is built for the inverse RL problem upon the state reconstruction technique. Given demonstrated inputs and outputs, a data-driven inverse Q -learning algorithm reconstructs the objective function without the knowledge of the demonstrated system dynamics or the OPFB gain. This algorithm yields unbiased solutions even though exploration noises exist. Convergence properties and the nonunique solution nature of the proposed algorithms are studied. Numerical simulation examples verify the effectiveness of the proposed methods.

This article studies the trajectory imitation control problem of linear systems suffering external disturbances and develops a data-driven static output feedback (OPFB) control-based inverse reinforcement learning (RL) approach. An Expert-Learner structure is considered where the learner aims to imitate expert's trajectory. Using only measured expert's and learner's own input and output data, the learner computes the policy of the expert by reconstructing its unknown value function weights and thus, imitates its optimally operating trajectory. Three static OPFB inverse RL algorithms are proposed. The first algorithm is a model-based scheme and serves as basis. The second algorithm is a data-driven method using input-state data. The third algorithm is a data-driven method using only input-output data. The stability, convergence, optimality, and robustness are well analyzed. Finally, simulation experiments are conducted to verify the proposed algorithms.

This article studies a distributed minmax strategy for multiplayer games and develops reinforcement learning (RL) algorithms to solve it. The proposed minmax strategy is distributed, in the sense that it finds each player’s optimal control policy without knowing all the other players’ policies. Each player obtains its distributed control policy by solving a distributed algebraic Riccati equation in a multiplayer noncooperative game. This policy is found against the worst policies of all the other players. We guarantee the existence of distributed minmax solutions and study their$\mathcal {L}_{2}$and asymptotic stabilities. Under mild conditions, the resulting minmax control policies are shown to improve robust gain and phase margins of multiplayer systems compared to the standard linear–quadratic regulator controller. Distributed minmax solutions are found using both model-based policy iteration and data-driven off-policy RL algorithms. Simulation examples verify the proposed formulation and its computational efficiency over the nondistributed Nash solutions.

This article proposes new inverse reinforcement learning (RL) algorithms to solve our defined Adversarial Apprentice Games for nonlinear learner and expert systems. The games are solved by extracting the unknown cost function of an expert by a learner using demonstrated expert's behaviors. We first develop a model-based inverse RL algorithm that consists of two learning stages: an optimal control learning and a second learning based on inverse optimal control. This algorithm also clarifies the relationships between inverse RL and inverse optimal control. Then, we propose a new model-free integral inverse RL algorithm to reconstruct the unknown expert cost function. The model-free algorithm only needs online demonstration of the expert and learner's trajectory data without knowing system dynamics of either the learner or the expert. These two algorithms are further implemented using neural networks (NNs). In Adversarial Apprentice Games, the learner and the expert are allowed to suffer from different adversarial attacks in the learning process. A two-player zero-sum game is formulated for each of these two agents and is solved as a subproblem for the learner in inverse RL. Furthermore, it is shown that the cost functions that the learner learns to mimic the expert's behavior are stabilizing and not unique. Finally, simulations and comparisons show the effectiveness and the superiority of the proposed algorithms.

In inverse reinforcement learning (RL), there are two agents. An expert target agent has a performance cost function and exhibits control and state behaviors to a learner. The learner agent does not know the expert's performance cost function but seeks to reconstruct it by observing the expert's behaviors and tries to imitate these behaviors optimally by its own response. In this article, we formulate an imitation problem where the optimal performance intent of a discrete-time (DT) expert target agent is unknown to a DT Learner agent. Using only the observed expert's behavior trajectory, the learner seeks to determine a cost function that yields the same optimal feedback gain as the expert's, and thus, imitates the optimal response of the expert. We develop an inverse RL approach with a new scheme to solve the behavior imitation problem. The approach consists of a cost function update based on an extension of RL policy iteration and inverse optimal control, and a control policy update based on optimal control. Then, under this scheme, we develop an inverse reinforcement Q-learning algorithm, which is an extension of RL Q-learning. This algorithm does not require any knowledge of agent dynamics. Proofs of stability, convergence, and optimality are given. A key property about the nonunique solution is also shown. Finally, simulation experiments are presented to show the effectiveness of the new approach.

static OPFB control problem of linear discrete-time (DT) systems. The primary contribution of the proposed algorithms lies in a newly developed OPFB control algorithm form for completely unknown systems. Under the premise of satisfying disturbance attenuation conditions, the conditions for the existence of the optimal OPFB solution are given. The convergence of the proposed Q -learning methods, and the difference and equivalence of two algorithms are rigorously proven. Moreover, considering the effects brought by probing noise for the persistence of excitation (PE), the proposed off-policy Q -learning method has the advantage of being immune to probing noise and avoiding biasedness of solution. Simulation results are presented to verify the effectiveness of the proposed approaches.

This work studies the${H}_{\infty }$-based minimal energy control with a preset convergence rate (PCR) problem for a class of disturbed linear time-invariant continuous-time systems with matched external disturbance. This problem aims to design an optimal controller so that the energy of the control input satisfies a predetermined requirement. Moreover, the closed-loop system asymptotic stability with PCR is ensured simultaneously. To deal with this problem, a modified game algebraic Riccati equation (MGARE) is proposed, which is different from the game algebraic Riccati equation in the traditional${H}_{\infty } $control problem due to the state cost being lost. Therefore, a unique positive-definite solution of the MGARE is theoretically analyzed with its existing conditions. In addition, based on this formulation, a novel approach is proposed to solve the actuator magnitude saturation problem with the system dynamics being exactly known. To relax the requirement of the knowledge of system dynamics, a model-free policy iteration approach is proposed to compute the solution of this problem. Finally, the effectiveness of the proposed approaches is verified through two simulation examples.

This article proposes robust inverse Q -learning algorithms for a learner to mimic an expert's states and control inputs in the imitation learning problem. These two agents have different adversarial disturbances. To do the imitation, the learner must reconstruct the unknown expert cost function. The learner only observes the expert's control inputs and uses inverse Q -learning algorithms to reconstruct the unknown expert cost function. The inverse Q -learning algorithms are robust in that they are independent of the system model and allow for the different cost function parameters and disturbances between two agents. We first propose an offline inverse Q -learning algorithm which consists of two iterative learning loops: 1) an inner Q -learning iteration loop and 2) an outer iteration loop based on inverse optimal control. Then, based on this offline algorithm, we further develop an online inverse Q -learning algorithm such that the learner mimics the expert behaviors online with the real-time observation of the expert control inputs. This online computational method has four functional approximators: a critic approximator, two actor approximators, and a state-reward neural network (NN). It simultaneously approximates the parameters of Q -function and the learner state reward online. Convergence and stability proofs are rigorously studied to guarantee the algorithm performance.

This article provides a novel inverse reinforcement learning (RL) algorithm that learns an unknown performance objective function for tracking control. The algorithm combines three steps: 1) an optimal control update; 2) a gradient descent correction step; and 3) an inverse optimal control (IOC) update. The new algorithm clarifies the relation between inverse RL and IOC. It is shown that the reward weight of an unknown performance objective that generates a target control policy may not be unique. We characterize the set of all weights that generate the same target control policy. We develop a model-based algorithm and, further, two model-free algorithms for systems with unknown model information. Finally, simulation experiments are presented to show the effectiveness of the proposed algorithms.

This article applies a singular perturbation theory to solve an optimal linear quadratic tracker problem for a continuous-time two-time-scale process. Previously, singular perturbation was applied for system regulation. It is shown that the two-time-scale tracking problem can be separated into a linear-quadratic tracker (LQT) problem for the slow system and a linear-quadratic regulator (LQR) problem for the fast system. We prove that the solutions to these two reduced-order control problems can approximate the LQT solution of the original control problem. The reduced-order slow LQT and fast LQR control problems are solved by off-policy integral reinforcement learning (IRL) using only measured data from the system. To test the effectiveness of the proposed method, we use an industrial thickening process as a simulation example and compare our method to a method with the known system model and a method without time-scale separation.

Current challenges in industrial processes control include achieving optimum operation for systems with two-time-scale dynamics and unknown models. This paper presents, for the first time, the integration of singular perturbation theory and reinforcement learning to solve this problem. To this end, an optimal operational control (OOC) problem with two time scales is formulated to reach the desired operational indices. Then, a singularly perturbed dynamics for two-time-scale industrial operational processes is developed by introducing a perturbed scale, resulting in the separation of the original system dynamics. Thus, the original optimization problem is decomposed into a reduced slow subproblem and a boundary fast subproblem. The fact that the sum of the separate solutions of these subproblems is approximately equal to the solution of the OOC problem is proven. Then, two Q-learning algorithms are proposed to obtain a composite feedback control. Finally, an industrial thickener example is employed to show the effectiveness of the proposed method.

Due to the interference issue in unlicensed band, sensor nodes frequently encounter degraded performance or lack of connection. This paper provides a real-time external interference source classification method for an 802.15.4-based wireless sensor network using deep learning. It uses RSSI sampling for collecting training data as well as online test data in an office environment. The output interference source type includes Wi-Fi beacon, different classes of WLAN traffic, BLE iBeacon, and microwave oven. Wireless sniffers are used to help labeling the ground truth of the sample data. We have trained a deep neural network with two hidden convolutional layers using raw RSSI samples as inputs. A micro-level model and a macro-level model are provided to predict the interference source based on the deep learning result. With implementation on both IEEE 802.15.4 SoC and Linux-based system, our experimental results show that the proposed framework can classify the major interference types with high accuracy.

Due to the unreliable nature of wireless links, sensor nodes frequently encounter degraded performance or lack of connection. This paper provides a real-time status monitoring and channel error diagnosis method in a wireless sensor network. It uses a poll and echo procedure to efficiently collect some physical layer and link layer statistics from both transmitter side and receiver side. A machine learning approach (k-nearest neighbor) is used for failure isolation. Our scheme involves learning the normal and anomalous behavior of the network via continued observation, and classifying future events and observations as normal or different classes of errors based on past experiences. With implementation on IEEE 802.15.4 SoC, our experimental results show that the proposed framework can diagnose the major wireless transmission errors with high accuracy.

The base station (BS) density configuration is a key factor to improve energy efficiency (EE) performances. In this paper, BS density configurations for achieving the optimal EE performance in two-tier cellular networks are analyzed, where the Poisson point process (PPP) is used to model the BS spatial distribution. To make the EE performances trackable and analyzable, the BS density optimization in each tier is transformed into an equivalent problem that jointly optimizes "the sum and the ratio of BS densities". This equivalent optimization problem is not necessarily convex, while its monotonicity with different power consumptions of BSs is analyzable. Considering the optimal BS density configuration is not unique and the closed-form solution for achieving the best EE performance is extremely difficult to be derived, a dynamic gradient based iterative algorithm by solving the quadratic functions is proposed. Furthermore, the quantitative analysis of EE performances based on the data fitting method shows that the approximately linear relationship between the optimal BS density and the user density holds only under specific conditions. Simulation results have demonstrated the effectiveness of the proposed algorithm and verified relevant conclusions.

Future mobile wireless communication networks will be featured as heterogeneity in order to enhance network performance and improve user experience. For better adaption to network challenges over its complexity and vulnerability, cell outage detection technique, a promising intelligent part of self-organizing networks (SON), has drawn considerable attention to deal with unexpected network faults. Our work is devoted to cell outage detection in a two-tier macro-pico network. Based on observation of performance metrics in time domain, we employ a classification algorithm called K-nearest neighbor (KNN) to achieve automatic anomaly detection. With some reasonable assumptions and a LTE-A system simulator, numerical experiments are implemented to demonstrate the efficiency of the proposed algorithm. Finally, localization for anomaly data and performance evaluation are further carried out to validate the classification accuracy.

With the rapid development of the mobile wireless system, the operator is experiencing unprecedented challenges on service maintenance and operational expenditure, which drives the demand for realizing automation in current networks. The cell outage detection is considered as an effective way to automatically detect network fault. Our work presents an automated cell outage detection mechanism in which a clustering technique called Dynamic Affinity Propagation (DAP) clustering algorithm is introduced. Performance metrics are collected from the network during its regular operation and then fed into the algorithm to produce optimal clusters for further anomaly detection. The proposed mechanism has been implemented in the LTE-Advanced simulation environment, through which we have successfully detected the configured cell outages and located their specific outage areas.