Abstract Multi-agent credit assignment is a research hotspot in the field of cooperative multi-agent reinforcement learning, and its key is how to accurately measure the individual contribution of each agent in the system to promote multi-agent cooperation. Existing solutions mainly use value function factorization or intrinsic reward mechanism, each of which has its own limitations, and both of them utilize global state information, which is not consistent with the information conditions in the actual confrontation. Therefore, this paper proposes a novel value factorization method for multi-agent deep reinforcement learning, which can solve the problem of credit assignment without using global state information. Our method establishes an explicit individual contribution evaluation mechanism for each agent, which portrays the role of each agent in the system by comparing the differences of joint value functions under different information conditions, so that more important agents get more attention, so as to improve the cooperative ability of agents. Experimental results show that our method outperforms all baselines in terms of learning efficiency and stability in multiple scenarios of StarCraft II, and its performance is comparable to that of the method based on global state information in easy scenarios.

Abstract Meta-learning is a pivotal and potentially influential machine learning approach to solve challenging problems in reinforcement learning. However, the costly hyper-parameter tuning for training stability of meta-learning is a known shortcoming and currently a hotspot of research. This paper addresses this shortcoming by introducing an online and easily trainable hyper-parameter optimization approach, called Meta Parameters Learning via Meta-Learning (MPML), to combine online hyper-parameter adjustment scheme into meta-learning algorithm, which reduces the need to tune hyper-parameters. Specifically, a basic learning rate for each training task is put forward. Besides, the proposed algorithm dynamically adapts multiple basic learning rate and a shared meta-learning rate through conducting gradient descent alongside the initial optimization steps. In addition, the sensitivity with respect to hyper-parameter choices in the proposed approach are also discussed compared with model-agnostic meta-learning method. The experimental results on reinforcement learning problems demonstrate MPML algorithm is easy to implement and delivers more highly competitive performance than existing meta-learning methods on a diverse set of challenging control tasks.

Abstract Value factorization is a popular method for cooperative multi-agent deep reinforcement learning. In this method, agents generally have the same ability and rely only on individual value function to select actions, which is calculated from total environment reward. It ignores the impact of individual characteristics of heterogeneous agents on actions selection, which leads to the lack of pertinence during training and the increase of difficulty in learning effective policies. In order to stimulate individual awareness of heterogeneous agents and improve their learning efficiency and stability, we propose a novel value factorization method based on Personality Characteristics, PCQMIX, which assigns personality characteristics to each agent and takes them as internal rewards to train agents. As a result, PCQMIX can generate heterogeneous agents with specific personality characteristics suitable for specific scenarios. Experiments show that PCQMIX generates agents with stable personality characteristics and outperforms all baselines in multiple scenarios of the StarCraft II micromanagement task.

Abstract In order to solve the incalculability caused by the issue of inconsistent objective functions in multi-agent deep reinforcement learning, the concept of Nash equilibrium is introduced. However, a Marko game may have multiple equilibriums, how to filter out a stable and optimal one is worth studying. Besides solution concept, how to keep the balance between exploration and exploitation is another key issue in reinforcement learning. On basis of the methods, which can converge to Nash equilibrium, this paper makes improvement through Pareto optimization. In order to alleviate the problem of over fitting caused by Pareto optimization and non-convergence caused by strategy change, we use stratified sampling in place of random sampling as assistance. What’s more, our methods are trained through fictitious self-play to make full of self-learning experiences. By analyzing the experiment carried out on MAgent platform, the proposed methods are not only far better than traditional methods, but also reaching or even surpassing the state of art MADRL methods.

Deep reinforcement learning (DRL) recently has attained remarkable results in various domains, including games, robotics, and recommender system. Nevertheless, an urgent problem in the practical application of DRL is fast adaptation. To this end, this article proposes a new and versatile metalearning approach called fast task adaptation via metalearning (FTAML), which leverages the strengths of the model-based methods and gradient-based metalearning methods for training the initial parameters of the model, such that the model is able to efficiently master unseen tasks with a little amount of data from the tasks. The proposed algorithm makes it possible to separate task optimization and task identification, specifically, the model-based learner helps to identify the pattern of a task, while the gradient-based metalearner is capable of consistently improving the performance with only a few gradient update steps through making use of the task embedding produced by the model-based learner. In addition, the choice of network for the model-based learner in the proposed method is also discussed, and the performance of networks with different depths is explored. Finally, the simulation results on reinforcement learning problems demonstrate that the proposed approach outperforms compared metalearning algorithms and delivers a new state-of-the-art performance on a variety of challenging control tasks.

Abstract Simple and efficient exploration remains a core challenge in deep reinforcement learning. While many exploration methods can be applied to high-dimensional tasks, these methods manually adjust exploration parameters according to domain knowledge. This paper proposes a novel method that can automatically balance exploration and exploitation, as well as combine on-policy and off-policy update targets through a dynamic weighted way based on value difference. The proposed method does not directly affect the probability of a selected action but utilizes the value difference produced during the learning process to adjust update target for guiding the direction of agent’s learning. We demonstrate the performance of the proposed method on CartPole-v1, MountainCar-v0, and LunarLander-v2 classic control tasks from the OpenAI Gym. Empirical evaluation results show that by integrating on-policy and off-policy update targets dynamically, this method exhibits superior performance and stability than does the exclusive use of the update target.

The balance between exploration and exploitation has always been a core challenge in reinforcement learning. This paper proposes “past-success exploration strategy combined with Softmax action selection”(PSE-Softmax) as an adaptive control method for taking advantage of the characteristics of the online learning process of the agent to adapt exploration parameters dynamically. The proposed strategy is tested on OpenAI Gym with discrete and continuous control tasks, and the experimental results show that PSE-Softmax strategy delivers better performance than deep reinforcement learning algorithms with basic exploration strategies.