As intelligent systems advance rapidly, human-robot collaboration is becoming increasingly important. Ensuring that the intelligent agent's behaviors match human intentions and value preferences is crucial for effective collaboration, which is termed the value alignment problem. Within the Reinforcement Learning (RL) paradigm, value alignment typically relies on pre-designed reward functions, and Cooperative Inverse Reinforcement Learning (CIRL) is often used to model value alignment as a human-robot game. However, existing works often assume that human is perfectly rational, and can fully obtain robot’s belief on human’s preference. To address this limitation, we propose a Particle Filter-based Hierarchical Dynamic Programming algorithm (PFHDP). By modeling the robot's belief state, this algorithm ensures the correct updates of human's estimate of the robot's belief. This allows human to adopt more targeted pedagogical behaviors to guide the robot based on her understanding of the robot's current belief, achieving belief alignment between human and robot and thereby promoting value alignment more effectively. Furthermore, we run experiments to evaluate the proposed method in two cooperative scenarios against some typical benchmark approaches. The experimental results show that our method can strengthen the alignment of belief states between human and robot, leading to enhanced value alignment.

In multi-agent systems, how to cooperate with each other to jointly confront opponent agents is a very important issue (i.e., multi-agent cooperative confrontation problem). In recent years, multi-agent reinforcement learning has developed rapidly and become a general paradigm for solving multi-agent decision-making problems, so it has naturally been used to solve multi-agent cooperative confrontation problems. However, existing works usually focus on the cooperation among multiple agents, making it difficult for agents to adapt to highly dynamic adversarial environments. To address this issue, in this paper, we propose a multi-agent reinforcement learning algorithm embedded with opponent modeling (MARLeOM). The opponent modeling module constructs multi-level opponent models using the environment model and recursive reasoning, and then mixes multi-level opponent models to enhance the representational capability. The multi-agent reinforcement learning part adopts CTDE mechanism and Actor-Critic framework. In the centralized training phase, Critic can obtain the observation and action information of all agents to guide the learning of Actor; in the decentralized execution phase, Actor makes decisions based on local information, which includes its own observation and the predicted opponent action based on the opponent’s observation using the opponent modeling module. The combination of opponent modeling and multi-agent reinforcement learning enables agents to learn optimal cooperative confrontation methods. Empirical experiments on multiple cooperative adversarial tasks demonstrate that MARLeOM can achieve more effective adaptation and better performance than baseline methods.