Multi-Agent Reinforcement Learning (MARL) has proven to be effective in learning cooperative policies, where agents learn decentralized policies, sharing the same network parameters, through centralized training. However, this parameter sharing can lead to similar behaviors among agents, hindering effective exploration. Existing multi-agent diversity methods that rely on the variational inference methods to differentiate agents may suffer from significant overfitting, which in turn hinders the exploration of new trajectories. To encourage multi-agent diversity and efficient exploration, we propose Active Exploration with Agent-Identity (AEAI), a novel exploration method, which maximizes the entropy over trajectories of different agents to promote sufficient exploration. Moreover, we derive a novel lower bound for the mutual information objective based on the successor features to align the directions of trajectories and agent identities in order to learn agent identity-conditioned policies. We combine these two items and integrate our method with existing MARL methods. We evaluate our proposed AEAI on challenging multi-agent tasks across various MARL benchmarks. Experimental results show that our method consistently outperforms existing state-of-the-art methods, highlighting its effectiveness in fostering diversity and improving exploration.

Large-Scale multi-UAV systems have significant advantages in enhancing the coverage and reliability of communication networks due to their flexible deployment capabilities. However, existing strategies in UAV-assisted communications primarily optimize bit-level throughput and energy efficiency, making it difficult to ensure effective information transmission under low SINR or complex channel conditions. To address issue, we introduce a new paradigm by incorporating semantic communication into UAV networks, and formulate the 3D UAV-BSs deployment problem with the goal of enhancing semantic fidelity. Furthermore, to tackle the challenges of large-scale multi-agent collaborative decision-making, this paper proposes a novel method which improves the traditional mean-field multi-agent deep deterministic policy gradient (MF-MADDPG), by combining with kernel density estimation (KDE) to model the neighborhood action distribution, enhancing the stability of the policy in continuous action spaces. A semantic-aware reward function is designed based on a representative metric of semantic fidelity, which guides the UAVs toward regions of higher semantic significance. Simulation results show that the proposed method outperforms existing strategies in terms of semantic transmission quality and training stability, demonstrating its application potential in large-scale semantic communication environments.

Cooperative Multi-Agent Reinforcement Learning (MARL) has drawn increasing interest in recent works due to its significant achievements. However, there are still some challenges impeding the learning of optimal cooperative policies, such as insufficient exploration. Prior works typically adopt mutual information-based methods to encourage exploration. However, this category of methods does not necessarily encourage agents to fully explore the joint behavior space. To address this limitation, we propose a novel objective based on learning a representation function with a Lipschitz constraint to maximize the traveled distances in the joint behavior space, encouraging agents to learn joint behaviors with large variations and leading to sufficient exploration. We further implement our method on top of QMIX. We demonstrate the effectiveness of our method by conducting experiments on the LBF, SMAC, and SMACv2 benchmarks. Our method outperforms previous methods in terms of final performance and state-action space exploration.

Recent works have increasingly focused on learning decentralized policies for agents as a solution to the scalability challenges in Multi-Agent Reinforcement Learning (MARL), where agents typically share the parameters of a policy network to make action decisions. However, this parameter sharing can impede efficient exploration, as it may lead to similar behaviors among agents. Different from previous mutual information-based methods that promote multi-agent diversity, we introduce a novel multi-agent exploration method called Trajectory Entropy Exploration (TEE). Our method employs a particle-based entropy estimator to maximize the entropy of different agents' trajectories in a contrastive trajectory representation space, resulting in diverse trajectories and efficient exploration. This entropy estimator avoids challenging density modeling and scales effectively in high-dimensional multi-agent settings. We integrate our method with MARL algorithms by deploying an intrinsic reward for each agent to encourage entropy maximization. To validate the effectiveness of our method, we test our method in challenging multi-agent tasks from several MARL benchmarks. The results demonstrate that our method consistently outperforms existing state-of-the-art methods.

In cooperative Multi-Agent Reinforcement Learning (MARL), agents that share policy network parameters often learn similar behaviors, which hinders effective exploration and can lead to suboptimal cooperative policies. Recent advances have attempted to promote multi-agent diversity by leveraging the Wasserstein distance to increase policy differences. However, these methods cannot effectively encourage diverse policies due to ineffective Wasserstein distance caused by the policy similarity. To address this limitation, we propose Wasserstein Contrastive Diversity (WCD) exploration, a novel approach that promotes multi-agent diversity by maximizing the Wasserstein distance between the trajectory distributions of different agents in a latent representation space. To make the Wasserstein distance meaningful, we propose a novel next-step prediction method based on Contrastive Predictive Coding (CPC) to learn distinguishable trajectory representations. Additionally, we introduce an optimized kernel-based method to compute the Wasserstein distance more efficiently. Since the Wasserstein distance is inherently defined for two distributions, we extend it to support multiple agents, enabling diverse policy learning. Empirical evaluations across a variety of challenging multi-agent tasks demonstrate that WCD outperforms existing state-of-the-art methods, delivering superior performance and enhanced exploration.

Policy network parameter sharing is a commonly used technique in advanced deep multi-agent reinforcement learning (MARL) algorithms to improve learning efficiency by reducing the number of policy parameters and sharing experiences among agents. Nevertheless, agents that share the policy parameters tend to learn similar behaviors. To encourage multi-agent diversity, prior works typically maximize the mutual information between trajectories and agent identities using variational inference. However, this category of methods easily leads to inefficient exploration due to limited trajectory visitations. To resolve this limitation, inspired by the learning of pre-trained models, in this paper, we propose a novel Contrastive Trajectory Representation (CTR) method based on learning distinguishable trajectory representations to encourage multi-agent diversity. Specifically, CTR maps the trajectory of an agent into a latent trajectory representation space by an encoder and an autoregressive model. To achieve the distinguishability among trajectory representations of different agents, we introduce contrastive learning to maximize the mutual information between the trajectory representations and learnable identity representations of different agents. We implement CTR on top of QMIX and evaluate its performance in various cooperative multi-agent tasks. The empirical results demonstrate that our proposed CTR yields significant performance improvement over the state-of-the-art methods.