Game theory plays a pivotal role in addressing Imperfect Information Games (IIGs), having given rise to the development of a suite of superhuman agents. However, constructing information sets and game trees in game theory approaches restricts the transferability of strategies. Recently, Reinforcement Learning (RL) has been introduced from the perfect information game to IIG, owing to its generalization capabilities. Yet, the challenges of efficient feature abstraction and stable learning within the RL paradigm persist. To address these challenges, we explore a stable actor-critic training framework with knowledge distillation based on supervised learning to enhance proximal policy optimization in strategy learning (MCA-PPO). For feature extraction, we engineer a multi-scale convolutional neural network that amalgamates local feature information across various receptive field sizes, unearthing the latent information embedded within potential feature sequences. We refine the network architecture further by integrating it with an attention mechanism, enabling it to discern relationships between feature channels and to autonomously identify the essence of multi-scale features. Regarding strategy learning, we employ the knowledge distillation technique to mitigate the instability inherent in proximal policy optimization, thereby hastening the progression of strategic learning. Finally, we conducted a systematic and comprehensive analysis of the MCA-PPO algorithm through a series of experiments on Teax Hold’em poker, showing the efficacy and transferability of the MCA-PPO framework.

Meta-Reinforcement Learning (Meta-RL) is a machine learning paradigm aimed at learning reinforcement learning policies that can quickly adapt to unseen tasks with few-shot data. Nevertheless, applying Meta-RL to real-world applications faces challenges due to the cost of data acquisition. To address this problem, offline Meta-RL has emerged as a promising solution, focusing on learning policies from pre-collected data that can effectively and rapidly adapt to unseen tasks. In this paper, we propose a new offline Meta-RL method called Meta-Actor-Critic with Evolving Gradient Agreement (MACEGA). MACEGA utilizes an evolutionary approach to estimate meta-gradients conductive to generalization across unseen tasks. During meta-training, gradient evolution is utilized to meta-update the value network and policies. Moreover, we use gradient agreement as an optimization objective for meta-learning, thereby enhancing the generalization ability of the meta-policy. We experimentally demonstrate the robustness of MACEGA in handling offline data quality. Furthermore, extensive experiments on various benchmarks provide empirical evidence that MACEGA outperforms previous state-of-the-art methods in generalizing to unseen tasks, thus demonstrating its potential for real-world applications.

Reinforcement learning (RL) has become a dominant decision-making paradigm and has achieved notable success in many real-world applications. Notably, deep neural networks play a crucial role in unlocking RL’s potential in large-scale decision-making tasks. Inspired by current major success of Transformer in natural language processing and computer vision, numerous bottlenecks have been overcome by combining Transformer with RL for decision-making. This paper presents a multiangle systematic survey of various Transformer-based RL (TransRL) models applied in decision-making tasks, including basic models, advanced algorithms, representative implementation instances, typical applications, and known challenges. Our work aims to provide insights into problems that inherently arise with the current RL approaches, and examines how we can address them with better TransRL models. To our knowledge, we are the first to present a comprehensive review of the recent Transformer research developments in RL for decision-making. We hope that this survey provides a comprehensive review of TransRL models and inspires the RL community in its pursuit of future directions. To keep track of the rapid TransRL developments in the decision-making domains, we summarize the latest papers and their open-source implementations at https://github.com/williamyuanv0/Transformer-in-Reinforcement-Learning-for-Decision-Making-A-Survey .

In cognitive radio systems, there are cases where malicious users transmit interference power to prevent secondary users from transmitting information. Due to the destructive behavior of malicious users, spectrum utilization efficiency will be reduced. The applications of game theory to study this confrontation relationship are reasonable. Based on the Continuous Blotto Game (CBG) model under the condition of one-shot perfect information confrontation, this paper constructs an anti-jamming game model for power allocation confrontation between secondary users and malicious users, and simulates the competition between two players under fixed resource constraints. An evolutionary learning approach is proposed, which improves the performance of the power allocation strategy through the repeated game of two players, and can obtain the same effect as the Nash equilibrium strategy under certain conditions. Our approach realizes the optimal power allocation on the information transmission channel in interference countermeasures without knowing the power allocation strategy of malicious users, thereby realizing the optimal utilization of spectrum resources. The simulation results show that, compared with the greedy algorithm and random algorithm, our algorithm has a more obvious effect on improving the spectrum utilization of secondary users.

Efficient and stable exploration remains a key challenge for deep reinforcement learning (DRL) operating in high-dimensional action and state spaces. Recently, a more promising approach by combining the exploration in the action space with the exploration in the parameters space has been proposed to get the best of both methods. In this article, we propose a new iterative and close-loop framework by combining the evolutionary algorithm (EA), which does explorations in a gradient-free manner directly in the parameters space with an actor-critic, and the deep deterministic policy gradient (DDPG) reinforcement learning algorithm, which does explorations in a gradient-based manner in the action space to make these two methods cooperate in a more balanced and efficient way. In our framework, the policies represented by the EA population (the parametric perturbation part) can evolve in a guided manner by utilizing the gradient information provided by the DDPG and the policy gradient part (DDPG) is used only as a fine-tuning tool for the best individual in the EA population to improve the sample efficiency. In particular, we propose a criterion to determine the training steps required for the DDPG to ensure that useful gradient information can be generated from the EA generated samples and the DDPG and EA part can work together in a more balanced way during each generation. Furthermore, within the DDPG part, our algorithm can flexibly switch between fine-tuning the same previous RL-Actor and fine-tuning a new one generated by the EA according to different situations to further improve the efficiency. Experiments on a range of challenging continuous control benchmarks demonstrate that our algorithm outperforms related works and offers a satisfactory trade-off between stability and sample efficiency.

We investigate a decentralized patrolling problem for dynamic environments where information is distributed alongside threats. In this problem, agents obtain information at a location, but may suffer attacks from the threat at that location. In a decentralized fashion, each agent patrols in a designated area of the environment and interacts with a limited number of agents. Therefore, the goal of these agents is to coordinate to gather as much information as possible while limiting the damage incurred. Hence, we model this class of problem as a transition-decoupled partially observable Markov decision process with health constraints. Furthermore, we propose scalable decentralized online algorithms based on Monte Carlo tree search and a factored belief vector. We empirically evaluate our algorithms on decentralized patrolling problems and benchmark them against the state-of-the-art online planning solver. The results show that our approach outperforms the state-of-the-art by more than 56% for six agents patrolling problems and can scale up to 24 agents in reasonable time.

Land cover classification for the evaluation of land cover changes over certain areas or time periods is crucial for geospatial modeling, environmental crisis evaluation and urban open space planning. Remotely sensed images of various spatial and spectral resolutions make it possible to classify land covers on the level of pixels. Semantic meanings of large regions consisting of hundreds of thousands of pixels cannot be revealed by discrete and individual pixel classes, but can be derived by integrating various groups of pixels using ontologies. This paper combines data of different resolutions for pixel classification by support vector classifiers, and proposes an efficient algorithm to group pixels based on classes of neighboring pixels. The algorithm is linear in the number of pixels of the target area, and is scalable to very large regions. It also re-evaluates imprecise classifications according to neighboring classes for region level semantic interpretations. Experiments on advanced spaceborne thermal emission and reflection radiometer (ASTER) data of more than six million pixels show that the proposed approach achieves up to 99.8% cross validation accuracy and 89.25% test accuracy for pixel classification, and can effectively and efficiently group pixels to generate high level semantic concepts.