In multi-agent reinforcement learning (MARL), independent learning (IL) often shows remarkable performance and easily scales with the number of agents. Yet, using IL can be inefficient and runs the risk of failing to successfully train, particularly in scenarios that require agents to coordinate their actions. Using centralised learning (CL) enables MARL agents to quickly learn how to coordinate their behaviour but employing CL everywhere is often prohibitively expensive in real-world applications. Besides, using CL in value-based methods often needs strong representational constraints (e.g. individual-global-max condition) that can lead to poor performance if violated. In this paper, we introduce a novel plug & play IL framework named Multi-Agent Network Selection Algorithm (MANSA) which selectively employs CL only at states that require coordination. At its core, MANSA has an additional agent that uses switching controls to quickly learn the best states to activate CL during training, using CL only where necessary and vastly reducing the computational burden of CL. Our theory proves MANSA preserves cooperative MARL convergence properties, boosts IL performance and can optimally make use of a fixed budget on the number CL calls. We show empirically in Level-based Foraging (LBF) and StarCraft Multi-agent Challenge (SMAC) that MANSA achieves fast, superior and more reliable performance while making 40% fewer CL calls in SMAC and using CL at only 1% CL calls in LBF.

Unmanned Combat Aerial Vehicles (UCAVs) is becoming a crucial platform to perform dangerous missions with the ability of autonomy and intelligence in making decisions. Recently, many researches have focused on UCAV air-to-air combat mission and attempted to solve maneuvering policy using deep reinforcement learning (DRL). Yet, previous studies on the sequential decision problem often have limitations due to the complexity of combat environment, overdependence on expert knowledge, and low learning efficiency from large-scale exploration space. In view of the lacks, the paper aims to automatically formulate counter maneuvers with the application of DRL and proposes a novel dynamic quality replay (DQR) method which is capable of guiding agent learn tactical policy from historical data efficiently, and can get rid of the dependence on traditional expert knowledge. Firstly, in order to accelerate model convergence, confrontation demonstrations generated with traditional algorithms are used for agent basic training before interacting with the environment. Moreover, the formal training is divided into four stages, in which DQR introduces a common framework to filter new interactive data for memory replay, including the process of classification, evaluation and sampling. The episodes are classified into different class based on opponent models and confrontation result. DQR can evaluate sample quality in each class and sample training batch with episode quality dynamically The new method is suitable for continuous action or discrete action space, and can makes pay-offs between specific demonstrations and exploration. Finally, we designed different initial engagement scenarios to train the agent models, whose basic algorithms are DQN, VPG, DDPG and SAC. In the test experiment, the winning rate of models can reach 0.6 or more, and the best model SAC can reach 0.8 for each stage, which means the feasibility and efficiency of the methods.