Geometric problem solving has always been a long-standing challenge in the fields of mathematical reasoning and artificial intelligence. We built a neural-symbolic system, called FGeo-HyperGNet, to automatically perform human-like geometric problem solving. The symbolic component is a formal system built on FormalGeo, which can automatically perform geometric relational reasoning and algebraic calculations and organize the solution into a hypergraph with conditions as hypernodes and theorems as hyperedges. The neural component, called HyperGNet, is a hypergraph neural network based on the attention mechanism, including an encoder to effectively encode the structural and semantic information of the hypergraph and a theorem predictor to provide guidance in solving problems. The neural component predicts theorems according to the hypergraph, and the symbolic component applies theorems and updates the hypergraph, thus forming a predict-apply cycle to ultimately achieve readable and traceable automatic solving of geometric problems. Experiments demonstrate the correctness and effectiveness of this neural-symbolic architecture. We achieved state-of-the-art results with a TPA of 93.50% and a PSSR of 88.36% on the FormalGeo7K dataset.

Despite significant advances in multi-agent reinforcement learning (MARL) in recent years, learning about cooperative policies for complex tasks remains a challenge. The concept of task allocation provides a valuable tool for designing and comprehending intricate multi-agent systems, as it enables agents with similar roles to collaborate on the same subtasks. However, existing research efforts have not yet provided clear guidelines on how task allocation can better adapt to dynamic environments. This paper introduces a novel framework called Hierarchical Cooperative Task Allocation (HCTA) that is based on long-term behavior analysis. The proposed framework is designed to solve the dynamic task allocation problem in fully cooperative tasks within multi-agent systems. Firstly, we train a task selector based on action chains to enhance the efficiency of task allocation. Secondly, we introduce a hierarchical cooperative mechanism to realize the parallel training of upper-level task selection and lower-level task decision making, so as to accelerate the learning of cooperative strategy. Experimental results show that HTCA significantly improves the learning efficiency of joint policy through dynamic task allocation, and enhances the success rate in complex cooperative tasks in StarCraft II micromanagement benchmark.

Multi-agent reinforcement learning (MARL) has been applied to many multi-agent team tasks, such as multi-robot swarm control. Distributional Value Function Factorization (DFAC) follows the Distributional-Individual-Global-Max (DIGM) principle, which forces the individual's optimal action to be in accordance with the optimal joint action at all times. However, this principle leads to grade inflation, which limits agents to exploring better strategies than before. We focus on this issue and propose a novel MARL method named dueling mixing distributional Q-learning with fully parameters (FDMIX). Firstly, a parametric individual value network generates an individual distribution function and a utility value function, while the fractions are obtained through a fraction proposal network. Secondly, the conversion mixing network obeys a new advantage-based DIGM principle to generate a joint distribution action value based on the global state. Finally, we incorporate an N-step return-based loss function to achieve stable and efficient training. Our extensive tests on the multiple-particle environment and StarCraft II show that our method performs better than state-of-the-art algorithms noticeably.

Abstract Unmanned surface vessels (USVs) have been fully used in the civilian and military fields in recent years, which dramatically expands protective capability and detection range. However, the marine environment's complexity and variability make that verification of various advanced USVs control algorithms face high costs and high risks. In this article, we present USVs‐Sim, a novel high‐fidelity general simulation platform for USVs autonomous navigation data generation and control strategy testing. USVs‐Sim is a collection of high‐level extensible modules that allows the rapid development and testing of USVs configurations and facilitates the construction of complex ocean scenarios. USVs‐Sim supports the steering or thrusting limits of USVs, as well as unique dynamics profiles. The platform can specify specific USVs sensor systems and change the time of day and weather conditions to generate robust data. USVs‐Sim facilitates training of deep‐learning algorithms by enabling data export from USVs sensors, including vision data, lidar, relative positions of ocean targets. Therefore, USVs‐Sim allows for the rapid prototyping, development, and testing of USVs autonomous control algorithms in a complex marine environment. In this article, we detail the general simulation platform and testing several representative USVs intelligent control algorithms on the platform.

In a mixed cooperative-competitive environment, each agent needs to learn heterogeneous strategies. The complex game relationship between heterogeneous agents causes difficulties for strategy learning. In this paper, we propose a graph-based multi-agent reinforcement learning method to simplify the learning process, named hierarchical heterogeneous graph multi-agent actor-critic (H2G-MAAC). The method first uses a hierarchical heterogeneous graph to model the hierarchical relationships among multiple heterogeneous agents. Then, it conducts representation learning for multiple agents with hierarchical graph attention networks. Finally, it learns heterogeneous strategies with multi-agent actor-critic. We conduct experiments in Predator-Prey games. The results indicate that the proposed method can simplify the learning process and outperforms existing state-of-the-art methods.

Abstract Autonomous obstacle avoidance control of unmanned surface vessels (USVs) in complex marine environments is always fundamental for its scientific search and detection. Traditional methods usually model USV motion and environments in a mathematical way that needs perceptual information. Unfortunately, it is difficult to provide sufficient perceptual information due to complex marine environments, resulting in inaccurate modeling. Reinforcement learning has recently enjoyed increasing popularity in the problem of obstacle avoidance since it can settle problems by partially observable environment information. However, the autonomous USV obstacle avoidance using reinforcement learning is still facing the challenge of designing appropriate reward functions under complex marine environments. To address these issues, we propose a prior knowledge‐based USV reinforcement learning obstacle avoidance algorithm. In this algorithm, an actor‐critic network is used as the main architecture of the algorithm, and prior knowledge‐based reward shaping used to design relevant reward function for USV obstacle avoidance. A standard USV based on a visual sensor is designed, and the state input of the algorithm is through USV's front vision sensor. We conducted simulation experiments and results prove that our algorithm can effectively converge, and USV achieves high velocity and low collision rate in the complex marine environment.

Learning adversarial policy aims to learn behavioural strategies for agents with different goals, is one of the most significant tasks in multi-agent systems. Multi-agent reinforcement learning (MARL), as a state-of-the-art learning-based model, employs centralised or decentralised control methods to learn behavioural strategies by interacting with environments. It suffers from instability and slowness in the training process. Considering that parallel simulation or computation is an effective way to improve training performance, we propose a novel MARL method called Multiple scenes multi-agent proximal Policy Optimisation (MPO) in this paper. In MPO, we first simulate multiple parallel scenes in the training environment. Multiple policies control different agents in the same scene, and each policy also controls several identical agents from multiple scenes. Then, we expand proximal policy optimisation (PPO) with an improved actor-critic network, ensuring the stability of training in multi-agent tasks. The actor network only uses local information for decision making, and the critic network uses global information for training. Finally, effective training trajectories are computed with two criteria from multiple parallel scenes rather than single to accelerate the learning process. We evaluate our approach in two simulated 3D environments, one of which is Unity's official open-source soccer game, and the other is unmanned surface vehicles (USVs) built by Unity. Experiments demonstrate that MPO converges more stable and faster than benchmark methods in model training, and demonstrates excellent adversarial policy compared with benchmark models.

Cooperation among agents with partial observation is an important task in multi-agent reinforcement learning (MARL), aiming to maximize a common reward. Most existing cooperative MARL approaches focus on building different model frameworks, such as centralized, decentralized, and centralized training with decentralized execution. These methods employ partial observation of agents as input directly, but rarely consider the local relationship between agents. The local relationship can help agents integrate observation information among different agents in a local range, and then adopt a more effective cooperation policy. In this paper, we propose a MARL method based on spatial relationship called hierarchical relation graph soft actor-critic (HRG-SAC). The method first uses a hierarchical relation graph generation module to represent the spatial relationship between agents in local space. Second, it integrates feature information of the relation graph through the graph convolution network (GCN). Finally, the soft actor-critic (SAC) is used to optimize agents' actions in training for compliance control. We conduct experiments on the Food Collector task and compare HRG-SAC with three baseline methods. The results demonstrate that the hierarchical relation graph can significantly improve MARL performance in the cooperative task.

Experience memory, which stores and replays experience from the past for off-policy reinforcement learning (RL) agents, has become an essential component of cutting-edge algorithms in recent years. In most previous work, experience memory emphasizes recently acquired experiences while neglects long-ago ones, which causes high variance and catastrophic forgetting in continual learning. To solve the problem, we introduce the Staged Experience Mechanism(SEM) - a novel management mechanism of experience memory. SEM adaptively regulates the proportion of experiences based on the current learning stages, enabling agents not only to learn from new experiences but also from very old ones given limited memory capacity. In the experiment, SEM is coupled with two state-of-the-art off-policy algorithms compared with classic experience replay mechanisms on the suite of MuJoCo tasks. The result shows that RL with SEM is more stable and less variable than its counterparts in most tested continuous control environments, which reduces the training variance by 5% ∼ 104% while ensuring optimal scoring performance.

Instability and slowness are two main problems in deep reinforcement learning. Even if proximal policy optimization (PPO) is the state of the art, it still suffers from these two problems. We introduce an improved algorithm based on proximal policy optimization, mixed distributed proximal policy optimization (MDPPO), and show that it can accelerate and stabilize the training process. In our algorithm, multiple different policies train simultaneously and each of them controls several identical agents that interact with environments. Actions are sampled by each policy separately as usual, but the trajectories for the training process are collected from all agents, instead of only one policy. We find that if we choose some auxiliary trajectories elaborately to train policies, the algorithm will be more stable and quicker to converge especially in the environments with sparse rewards.