ACyber-Physical System(CPS) is a system which consists of software components and physical components. Traditional system verification techniques such as model checking or theorem proving are difficult to apply to CPS because the physical components have infinite number of states. To solve this problem, robustness guided falsification of CPS is introduced. Robustness measures how robustly the given specification is satisfied. Robustness guided falsification tries to minimize the robustness by changing inputs and parameters of the system. The input with a minimal robustness (counterexample) is a good candidate to violate the specification. Existing methods use several optimization techniques to minimize robustness. However, those methods do not use temporal structures in a system input and often require a large number of simulation runs to minimize the robustness. In this paper, we explore state-of-the-artDeep Reinforcement Learning(DRL) techniques, i.e.,Asynchronous Advantage Actor-Critic(A3C) andDouble Deep Q Network(DDQN), to reduce the number of simulation runs required to find such counterexamples. We theoretically show how robustness guided falsification of a safety property is formatted as a reinforcement learning problem. Then, we experimentally compare the effectiveness of our methods with three baseline methods, i.e., random sampling, cross entropy and simulated annealing, on three well known CPS systems. We thoroughly analyse the experiment results and identify two factors of CPS which make DRL based methods better than existing methods. The most important factor is the availability of the system internal dynamics to the reinforcement learning algorithm. The other factor is the existence of learnable structure in the counterexample.

Reinforcement learning agents usually learn from scratch, which requires a large number of interactions with the environment. This is quite different from the learning process of human. When faced with a new task, human naturally have the common sense and use the prior knowledge to derive an initial policy and guide the learning process afterwards. Although the prior knowledge may be not fully applicable to the new task, the learning process is significantly sped up since the initial policy ensures a quick-start of learning and intermediate guidance allows to avoid unnecessary exploration. Taking this inspiration, we propose knowledge guided policy network (KoGuN), a novel framework that combines human prior suboptimal knowledge with reinforcement learning. Our framework consists of a fuzzy rule controller to represent human knowledge and a refine module to finetune suboptimal prior knowledge. The proposed framework is end-to-end and can be combined with existing policy-based reinforcement learning algorithm. We conduct experiments on several control tasks. The empirical results show that our approach, which combines suboptimal human knowledge and RL, achieves significant improvement on learning efficiency of flat RL algorithms, even with very low-performance human prior knowledge.

Many methods for reinforcement learning use symbolic representations-nonemergent-such as Q-learning. We use emergent representations here, without human handcrafted symbolic states (i.e., each state corresponds to a different location). This paper models reinforcement learning for hidden neurons in emergent networks for sequential tasks. In this paper, their influences on sequential tasks (e.g., robot navigation in different scenarios) are investigated where the learned value and results of a behavior rely on not only the current experience just like in a pattern recognition (episodic) but also the prediction of future experiences (e.g., delayed rewards) and environments (e.g., previously learned navigational trajectories). We show that this new model of motivated learning amounts to the computation of the maximum-likelihood estimate through "life" where punishment and reward have increased weights. This new formulation avoids the greediness of time-discount in Q-learning. Its complex nonlinear sequential optimization has been solved in a closed-form procedure under the condition of the limited computational resources and limited learning experience so far, because we convert it into a simpler problem of incremental and linear estimation. The experimental results showed that the serotonin and dopamine systems speed up learning for sequential tasks, because not all events are equally important. As far as we know, this is the first work that studies the influences of reinforcers (via serotonin and dopamine) on hidden neurons (Y neurons) for sequential tasks in dynamic scenarios using emergent representations.

Serotonin and dopamine transmitters are synthesized in the lower brain but are transmitted widely to many areas of the brain. Emergent representations are critical in understanding their effects. In our prior work [26], their effects on internal, non-motor neurons are studied for pattern recognition tasks only. In this paper, we study their effects on sequential tasks - robot navigation with different settings. They are sequential tasks because the outcome of behavior depends on not only the current behavior as in pattern recognition but also the previous behaviors (e.g., previous navigational trajectories). Analytically, we show that the serotonin and dopamine systems affect the performance of sequential tasks in a compounded way. Experimentally, we show that the effect on the learning rate of feature neurons (in the Y area) allows the agent to approach the friend and avoid the enemy faster as compounding effects of sequential states. Further, we tested the effect of punishment and reward schedule with the same initial locations. We also experimented the effect of punishment and reward schedule with random initial locations. These experiments all indicated that the reinforcement learning via the serotonin and the dopamine systems is beneficial for developing desirable behaviors in this set of sequential tasks - staying close to its friend and away from its enemy. As far as we know, this is the first work that investigates the effects of reinforcer (via serotonin and dopamine) on internal neurons for sequential tasks.