Accelerating learning processes for complex tasks by leveraging previously learned tasks has been one of the most challenging problems in reinforcement learning, especially when the similarity between source and target tasks is low. This work proposes REPresentation And INstance Transfer (REPAINT) algorithm for knowledge transfer in deep reinforcement learning. REPAINT not only transfers the representation of a pre-trained teacher policy in the on-policy learning, but also uses an advantage-based experience selection approach to transfer useful samples collected following the teacher policy in the off-policy learning. Our experimental results on several benchmark tasks show that REPAINT significantly reduces the total training time in generic cases of task similarity. In particular, when the source tasks are dissimilar to, or sub-tasks of, the target tasks, REPAINT outperforms other baselines in both training-time reduction and asymptotic performance of return scores.

DeepRacer is a platform for end-to-end experimentation with RL and can be used to systematically investigate the key challenges in developing intelligent control systems. Using the platform, we demonstrate how a 1/18th scale car can learn to drive autonomously using RL with a monocular camera. It is trained in simulation with no additional tuning in the physical world and demonstrates: 1) formulation and solution of a robust reinforcement learning algorithm, 2) narrowing the reality gap through joint perception and dynamics, 3) distributed on-demand compute architecture for training optimal policies, and 4) a robust evaluation method to identify when to stop training. It is the first successful large-scale deployment of deep reinforcement learning on a robotic control agent that uses only raw camera images as observations and a model-free learning method to perform robust path planning. We open source our code and video demo on GitHub2.

In this work, we study the problem of multi-agent reinforcement learning (MARL) with model uncertainty, which is referred to as robust MARL. This is naturally motivated by some multi-agent applications where each agent may not have perfectly accurate knowledge of the model, e.g., all the reward functions of other agents. Little a priori work on MARL has accounted for such uncertainties, neither in problem formulation nor in algorithm design. In contrast, we model the problem as a robust Markov game, where the goal of all agents is to find policies such that no agent has the incentive to deviate, i.e., reach some equilibrium point, which is also robust to the possible uncertainty of the MARL model. We first introduce the solution concept of robust Nash equilibrium in our setting, and develop a Q-learning algorithm to find such equilibrium policies, with convergence guarantees under certain conditions. In order to handle possibly enormous state-action spaces in practice, we then derive the policy gradients for robust MARL, and develop an actor-critic algorithm with function approximation. Our experiments demonstrate that the proposed algorithm outperforms several baseline MARL methods that do not account for the model uncertainty, in several standard but uncertain cooperative and competitive MARL environments.

As industrial systems become more complex with increasing connectivity, there has been a growing demand for scallable and robust methods for identification of hundreds to thousands system parameters from large time-series data (i.e., ordered measurements). In the past, Artificial Neural Networks (ANNs) have been used successfully in system identification and controls of various systems such as magnetic levitation. Recent literature revealed how ANNs can also be used directly for extracting parametric information from dynamic systems. However, scalability of these approaches to complex systems, their robustness to sensor noise have not been well-studied for system identification of closed-loop control systems. This paper addresses the scalability and robustness issues in complex nonlinear system identification of dynamic system within the framework of deep convolutional neural networks (CNNs). CNNs provide means to model local receptive fields, shared weights, and spatial or temporal subsampling, and some of the robustness issues. In addition, CNN weights are learned with back-propagation with respect to minimizing a loss function. Therefore, the weights are synthesized similar to traditional prediction error methods without the need to manually configure hundreds and thousands of ordinary differential equations. In this paper, a new deep CNN architecture to overcome scalability and robustness issues is proposed. The new deep CNN architecture is implemented in Keras with Tensorflow backend, and applied to building energy load prediction problem.

This paper studies online fault detection and isolation of modular dynamic systems modeled as sets of place-bordered Petri nets. The common places among the set of Petri nets modeling a system capture coupling of various system components. The transitions are labeled by events, some of which are unobservable (i.e., not directly recorded by the sensors attached to the system). The events whose occurrence must be diagnosed have unobservable transition labels. These events model faults or other significant changes in the system state. The existing theory of diagnosis of discrete-event systems is extended in the context of the above model. The modular structure of the system is exploited by a distributed algorithm for fault diagnosis. A Petri net diagnoser is associated with every Petri net and the diagnosers communicate in real time during the diagnostic process when the token count of common places changes. A merge function is defined to combine the individual diagnoser states and recover the complete diagnoser state that would be obtained under a monolithic approach. Strategies that reduce the communication overhead are presented. The software implementation of the distributed algorithm is discussed. Note to Practitioners-In the last decade, monitoring, fault detection, and diagnosis methodologies based on the use of discrete-event models have been successfully used in a variety of technological systems ranging from document processing systems to intelligent transportation systems. This paper was motivated by the problem of fault diagnosis for modular (distributed) dynamic discrete-event systems (DES). As a DES modeling formalism, Petri nets offer potential advantages in terms of the distributed representation of the system and the ability to represent coupling of the system components. The systems studied in this paper are sets of modules coupled with each other through various system components and modeled using Petri nets. We present a distributed fault diagnosis algorithm which allows each module in the distributed system to diagnose its faults independently unless completion of a task requires the use of coupled components. In the case of coupling, modules communicate with each other to accurately diagnose the fault. The distributed fault diagnosis algorithm recovers the monolithic diagnosis information at the cost of communication and growing communication overhead. To mitigate that problem, we present an improved version of the algorithm that significantly reduces the communication overhead. Finally, we introduce the software toolbox (written in Matlab and integrated with AT&T Graphviz) and we present a case study of an example of a heating, ventilation, and air-conditioning system where we use the software tool for modeling and analyzing the system