Black-box artificial intelligence (AI) induction methods such as deep reinforcement learning (DRL) are increasingly being used to find optimal policies for a given control task. Although policies represented using a black-box AI are capable of efficiently executing the underlying control task and achieving optimal closed-loop performance-controlling the agent from the initial time step until the successful termination of an episode, the developed control rules are often complex and neither interpretable nor explainable. In this article, we use a recently proposed nonlinear decision-tree (NLDT) approach to find a hierarchical set of control rules in an attempt to maximize the open-loop performance for approximating and explaining the pretrained black-box DRL (oracle) agent using the labeled state-action dataset. Recent advances in nonlinear optimization approaches using evolutionary computation facilitate finding a hierarchical set of nonlinear control rules as a function of state variables using a computationally fast bilevel optimization procedure at each node of the proposed NLDT. In addition, we propose a reoptimization procedure for enhancing the closed-loop performance of an already derived NLDT. We evaluate our proposed methodologies (open- and closed-loop NLDTs) on different control problems having multiple discrete actions. In all these problems, our proposed approach is able to find relatively simple and interpretable rules involving one to four nonlinear terms per rule, while simultaneously achieving on par closed-loop performance when compared to a trained black-box DRL agent. A postprocessing approach for simplifying the NLDT is also suggested. The obtained results are inspiring as they suggest the replacement of complicated black-box DRL policies involving thousands of parameters (making them noninterpretable) with relatively simple interpretable policies. The results are encouraging and motivating to pursue further applications of proposed approach in solving more complex control tasks.

In this paper, a unified batch-online learning approach is introduced to learn a linear representation of nonlinear system dynamics using the Koopman operator. The presented system modeling approach leverages a novel incremental Koopman-based update law that regains a mini-collection of samples stored in a memory to minimize not only the instantaneous Koopman operator’s identification errors but also the identification errors for the collection of retrieved samples. Discontinuous modifications of gradient flows are presented for the online update law to assure finite-time convergence under easy-to-verify conditions defined on the batch of data. Therefore, this unified online-batch framework allows joint sample- and time-domain analysis to converge the Koopman operator’s parameters. More specifically, it is shown that if the collected mini-batch of samples guarantees a rank condition, then finite-time guarantee in the time domain can be certified, and the settling time depends on the quality of collected samples being reused in the update law. Moreover, the efficiency of the proposed Koopman-based update law is further analyzed by showing that the identification regret in continuous time grows sub-linearly with time. Furthermore, to avoid learning corrupted dynamics due to the selection of an inappropriate set of Koopman observables, a higher-layer meta-learner employs a discrete Bayesian optimization algorithm to obtain the best library of observable functions for the operator. Since finite-time convergence of the Koopman model for each set of observables is guaranteed under a rank condition on stored data, the fitness of each set of observables can be obtained based on the identification error on the stored samples in the proposed framework and even without implementing any controller based on the learned system. Finally, to confirm the effectiveness of the proposed scheme, two simulation examples are presented.

In this paper, a three-level decision-making framework is developed to generate safe and effective decisions for autonomous vehicles (AVs). A key component in this decision framework is a normal-form game to capture the interactions between the ego vehicle and its surrounding vehicles. The payoffs in the normal-form game are designed to capture both safety reward and the reward gained by obeying (or the price paid by violating) “soft” traffic rules, e.g., first-come-first-go. This game formulation enables the ego to 1) make appropriate decisions considering the payoffs and possible actions of its surrounding vehicles, and 2) take intelligent actions in emergencies that may sacrifice some soft traffic rules to ensure safety. Moreover, we introduce parameters in the payoff matrix to tune the ego vehicle’s behavior, e.g., aggressiveness level. A neural network is developed to learn the tuning parameters via supervised learning. In addition, to enable the ego to respond timely to different surrounding vehicles’ driving styles, driving style characterization is incorporated into the payoff design for the normal-form game. Simulation studies are conducted to demonstrate the performance of the developed algorithms in two-vehicle intersection-crossing and lane-changing scenarios.

An online evolving method, named evolving finite state machine (e-FSM), is proposed to develop an optimal Markov driving model. The model has the same properties as a standard Markov model, but its states and transition dynamics evolve without human supervision. In this article, we introduce: 1) the principles of the e-FSM’s novel capabilities:online state determinationandonline transition-dynamics identificationfor elaborating the Markov driving model and 2) an advanced online evolving framework (a-OEF) for supporting the reinforcement-learning-based controller’s decision making by using the evolved model. For the evaluation of the proposed methodology and framework, the ego vehicle is controlled by the double deep${Q}$-network (DDQN) controller with and without the a-OEF in the multilane driving scenario where various naturalistic traffic situations are simulated. Simulation results show that better control performance in terms offastandsafedriving is achieved via the DDQN with the a-OEF, which demonstrates that the Markov driving models evolved by the e-FSMs effectively support detecting and revising the controller’s incorrect decision making.

This article describes a novel approach to the problem of developing explainable machine learning models. We consider a deep reinforcement learning (DRL) model representing a highway path planning policy for autonomous highway driving [1]. The model constitutes a mapping from the continuous multidimensional state space characterizing vehicle positions and velocities to a discrete set of actions in longitudinal and lateral direction. It is obtained by applying a customized version of the double deep Q-network learning algorithm [2]. The main idea is to approximate the DRL model with a set of IF-THEN rules that provide an alternative interpretable model, which is further enhanced by visualizing the rules. This concept is rationalized by the universal approximation properties of the rule-based models with fuzzy predicates. The proposed approach includes a learning engine composed of zero-order fuzzy rules, which generalize locally around the prototypes by using multivariate function models. The adjacent (in the data space) prototypes, which correspond to the same action, are further grouped and merged into the so-called MegaClouds reducing significantly the number of fuzzy rules. The input selection method is based on ranking the density of the individual inputs. Experimental results show that the specific DRL agent can be interpreted by approximating with families of rules of different granularity. The method is computationally efficient and can be potentially extended to addressing the explainability of the broader set of fully connected deep neural network models.

In this paper, we present a safe deep reinforcement learning system for automated driving. The proposed framework leverages merits of both rule-based and learning-based approaches for safety assurance. Our safety system consists of two modules namely handcrafted safety and dynamically-learned safety. The handcrafted safety module is a heuristic safety rule based on common driving practice that ensure a minimum relative gap to a traffic vehicle. On the other hand, the dynamically-learned safety module is a data-driven safety rule that learns safety patterns from driving data. Specifically, the dynamically-leaned safety module incorporates a model lookahead beyond the immediate reward of reinforcement learning to predict safety longer into the future. If one of the future states leads to a near-miss or collision, then a negative reward will be assigned to the reward function to avoid collision and accelerate the learning process. We demonstrate the capability of the proposed framework in a simulation environment with varying traffic density. Our results show the superior capabilities of the policy enhanced with dynamically-learned safety module.

In many industrial applications, one of the major bottlenecks in using advanced learning-based methods (such as reinforcement learning) for controls is the lack of interpretability of the trained agent. In this paper, we present a methodology for translating a trained reinforcement learning agent into a set of simple and easy to interpret if-then rules by using the proven universal approximation property of the rules with fuzzy predicates. Proposed methodology combines the optimality of reinforcement learning with interpretability of the theory of approximate reasoning, thus making reinforcement learning-based solutions more accessible to industrial practitioners. The framework presented in this paper has the potential to help address the fundamental problem in widespread adoption of reinforcement learning in industrial applications.

This paper describes a new self-organizing neuro-fuzzy approach to autonomously learn interpretable models by self-driving cars. A new explainable self-organizing architecture and a new density-based feature selection method are proposed. These new approaches are used to classify different action states occurring from different self-driving conditions. The proposed approach is able to provide human understandable IF ... THEN rules representation due to its learning engine which is composed of a massively parallel set of 0-order fuzzy rules. The proposed density-based feature selection method is based on the ranking of the densities of each feature in the data space, and takes advantage of the parallel characteristic of the proposed explainable self-organizing approach to create individualized subsets of features per class. The main goal of both proposed methods is to provide highly accurate models with high transparency, interpretability, and explainability for self-driving vehicles. In order to validate our proposal, experiments were realized using a real dataset provided by Ford Motor Company. The dataset contains different driving states occurring during self-driving performances. Results demonstrate that the proposed approach could surpass its state-of-the-art competitors in terms of accuracy for this challenge multiclass classification problem.

Deep reinforcement learning (DRL) techniques have been used to solve a discretionary lane change decision-making problem and are showing promising results. However, since the input information for the discretionary lane change problem is continuous and can be in high dimension, it is an open challenge for DRL to optimize the exploration-exploitation trade-off. Conventional model-less exploration methods lack a systematic way to incorporate additional engineering or model-based knowledge of our application into consideration and as a result, the training can be inefficient and may dwell on a policy, e.g. lane change strategy that is impractical. In previous related work, many used the rule-based safety check policy to guide the exploration and collect input information data. However, it is not guaranteed to get the optimal policy and the performance is dependent on the safety check policy selected. In this paper, we developed an explicit statistical aggregated environment model using a conditional variational auto-encoder and a model-based exploration strategy leveraging it. The agent is guided to explore with surprise-based intrinsic reward derived from the environment model. The result is compared with annealing epsilon-greedy exploration and with rule-based safety check exploration. We demonstrate that the performance of the developed model-based exploration method is comparable with the best rule-based safety check exploration and much better than the epsilon-greedy exploration.

The operational space of an autonomous vehicle (AV) can be diverse and vary significantly. Due to this, formulating a rule based decision maker for selecting driving maneuvers may not be ideal. Similarly, it may not be efficient to solve optimal control problem in real-time for a predefined cost function. In order to address these issues and to avoid peculiar behaviors when encountering unforeseen scenario, we propose a reinforcement learning (RL) based method, where the ego car, i.e., an autonomous vehicle, learns to make decisions by directly interacting with the simulated traffic. Here the decision maker is a deep neural network that provides an action choice for a given system state. We demonstrate the performance of the developed algorithm in highway driving scenario where the trained AV encounters varying traffic density.

Passivity-based control (PBC) for port-Hamiltonian systems provides an intuitive way of achieving stabilization by rendering a system passive with respect to a desired storage function. However, in most instances the control law is obtained without any performance considerations and it has to be calculated by solving a complex partial differential equation (PDE). In order to address these issues we introduce a reinforcement learning (RL) approach into the energy-balancing passivity-based control (EB-PBC) method, which is a form of PBC in which the closed-loop energy is equal to the difference between the stored and supplied energies. We propose a technique to parameterize EB-PBC that preserves the systems's PDE matching conditions, does not require the specification of a global desired Hamiltonian, includes performance criteria, and is robust. The parameters of the control law are found by using actor-critic (AC) RL, enabling the search for near-optimal control policies satisfying a desired closed-loop energy landscape. The advantage is that the solutions learned can be interpreted in terms of energy shaping and damping injection, which makes it possible to numerically assess stability using passivity theory. From the RL perspective, our proposal allows for the class of port-Hamiltonian systems to be incorporated in the AC framework, speeding up the learning thanks to the resulting parameterization of the policy. The method has been successfully applied to the pendulum swing-up problem in simulations and real-life experiments.