The performance of models trained by Multi-Agent Reinforcement Learning (MARL) is sensitive to perturbations in observations, lowering their trustworthiness in complex environments. Adversarial training is a valuable approach to enhance their performance robustness. However, existing methods often overfit to adversarial perturbations of observations and fail to incorporate prior information about the policy adopted by their protagonist agent, i.e., the primary one being trained. To address this important issue, this paper introduces Adversarial Training with Stochastic Adversary (ATSA), where the proposed adversary is trained online alongside the protagonist agent. The former consists of Stochastic Director (SDor) and SDor-guided generaTor (STor). SDor performs policy perturbations by minimizing the expected team reward of protagonists and maximizing the entropy of its policy, while STor generates adversarial perturbations of observations by following SDor's guidance. We prove that SDor's soft policy converges to a global optimum according to factorized maximum-entropy MARL and leads to the optimal adversary. This paper also introduces an SDor-STor loss function to quantify the difference between a) perturbations in the agent's policy and b) those advised by SDor. We evaluate our ATSA on StarCraft II tasks and autonomous driving scenarios, demonstrating that a) it is robust against diverse perturbations of observations while maintaining outstanding performance in perturbation-free environments, and b) it outperforms the state-of-the-art methods.

Reinforcement Learning (RL) is widely used in tasks where agents interact with an environment to maximize rewards. Building on this foundation, Safe Reinforcement Learning (Safe RL) incorporates a cost metric alongside the reward metric, ensuring that agents adhere to safety constraints during decision-making. In this paper, we identify that Safe RL is vulnerable to backdoor attacks, which can manipulate agents into performing unsafe actions. First, we introduce the relevant concepts and evaluation metrics for backdoor attacks in Safe RL. It is the first attack framework in the Safe RL field that involves both Positive and Negative Action sample (PNAct) is to implant backdoors, where positive action samples provide reference actions and negative action samples indicate actions to be avoided. We theoretically point out the properties of PNAct and design an attack algorithm. Finally, we conduct experiments to evaluate the effectiveness of our proposed backdoor attack framework, evaluating it with the established metrics. This paper highlights the potential risks associated with Safe RL and underscores the feasibility of such attacks. Our code and supplementary material are available at https://github.com/azure-123/PNAct.

Industry 5.0 enhances manufacturing ability through efficient human-machine interaction, combining human resources and robots to complete tasks more accurately and effectively. Artificial intelligence (AI) plays an essential role in Industry 5.0. As a branch in AI, multiagent deep reinforcement learning (MADRL) attracts vast attention in both academia and industry. However, there is a gap between virtual and physical environments in terms of howcleanan observed state is. In addition, state adversarial attacks can seriously impact the performance of MADRL. Hence, how to improve the robustness of MADRL algorithms is an important research topic. In this article, we propose an optimal policy-based state adversary attack method that would make the MADRL algorithm more robust when it is applied in the training process of agents. Two case studies related to Industry 5.0 and a general case study are presented in which robustness training against the optimal adversarial attack is tested. The MADRL algorithms involved in the experiments include centralized training and decentralized execution (CTDE) framework and shared experience actor-critic (SEAC) to demonstrate the universality of our method.

Abstract A transmission power minimization problem for a reconfigurable intelligent surface (RIS) aided non‐orthogonal multiple access (NOMA) network has been studied here, which has assumed the channel state information is imperfect. The channel error in this paper has been modeled as two different error models, that is, bounded error model and statistical error model. First, a two‐step algorithm is proposed to decouple the decoding order and optimization variables under bounded error model. Then, for the statistical error model, the alternating optimization method is adopted to decouple optimization variables. Finally, the performance difference is compared between the proposed beamforming scheme and the non‐RIS scheme from two key parameters, the number of reflected elements and the error factor. Simulation results verify that passive RIS has the advantage on the system performance compared with the scheme without RIS.

Multiagent deep reinforcement learning (MADRL) has been recently applied in many fields, including industry 5.0, but it is sensitive to adversarial attacks. Although adversarial attacks can be detrimental, they are crucial for testing and assisting in enhancing the robustness of models. Existing attacks on MADRL-based models are not sufficient since these attacks involve fixed perturbed agents, without taking into account cases where perturbed agents change. In this article, we present a novel adversarial attack framework. In this framework, we define critical agents that change over time, i.e., when they are perturbed a little, the whole multiagent system is perturbed greatly. Then, we identify critical agents through their worst-case joint actions. In this identifying process, we use gradient information, differential evolution, and SARSA to deal with the challenge caused by changes in the perturbed agents and to compute the worst-case joint actions. After identifying them, we use the target attack method to perturb them. We apply our method to attack the models trained by two state-of-the-art MADRL algorithms under three environments, including two industry-related ones. The experimental results demonstrate our method has a stronger perturbing ability than the existing methods.

In this paper, we investigate the massive multi-input multi-output (MIMO) transmission for the satellite communication systems equipped with a uniform rectangular array (URA) and aim to design the precoder with broad coverage radiation power pattern to improve the performance of time and frequency synchronizations. The modified Cramér-Rao vector bounds (MCRVB) are chosen as the benchmark of symbol timing offset and frequency offset estimation problem, which can be viewed as the time and frequency synchronization performance metric for the broad coverage precoder design. By considering the minimax MCRVB criterion and the per-antenna equal power constraint, the precoder design approach is formulated as the non-convex constrained minimax problem over the discrete radiation power pattern. This optimization problem is solved by the smoothing techniques combined with the manifold optimization method. To reduce the calculation complexity, the nonmonotone conjugate gradient method is applied. Simulation results show that the average and the minimum received powers in the coverage area of the proposed scheme are higher than those of the scheme based on fairness criterion. Compared with the existing omnidirectional and broad coverage schemes, the proposed scheme has the best synchronization performance among all the contrast solutions.

In this paper, we investigate broad coverage pre-coder design for 3D massive multi-input multi-output (MIMO) systems. We focus on the synchronization performance in the cell coverage. The log-distance path loss model in the line-of-sight (LoS) scenario is formulated. The synchronization performance can be characterized by the missed detection (MD) probability. By considering the equal MD probability in the cell, we formulate a criterion for the precoder design. Moreover, we consider the equal transmit power constraint on each antenna to efficiently utilize the power amplifier (PA) capacity of the BS. By using the manifold optimization framework, we design the precoder under the aforementioned criterion and constraint. Simulation results show that the fairness among all the users in this cell can be ensured. Compared with the precoders designed by half power beam-width (HPBW), the proposed scheme causes much less inter-cell interference, and has a better synchronization performance.

In this paper, we investigate broad coverage precoder design for 3D massive multi-input multi-output (MIMO) systems, where the base station (BS) is equipped with a uniform rectangular array (URA). We focus on the synchronization performance in the cell coverage. The flat fading channel model is formulated. The synchronization performance can be characterized by the missed detection (MD) probability. By considering the MD probability fairness in the cell, we formulate a criterion for the precoder design. Moreover, we consider the equal transmit power constraint on each antenna to efficiently utilize the power amplifier (PA) capacity of the BS. By using the manifold optimization framework, we design the precoder under the aforementioned criterion and constraint. Simulation results show that the fairness among all the users in this cell can be ensured. Compared with the precoders designed by half power beam-width (HPBW), the proposed scheme causes much less inter-cell interference and has a better synchronization performance.

In this paper, we design the precoding matrix with broad coverage for massive multi-input multi-output public channels. In order to guarantee the efficient use of power amplifiers and improve the achievable ergodic rate, equal transmit power per-antenna and semi-unitary constraints on the precoding matrix are considered simultaneously. Within the framework of manifold optimization, the precoding matrix design under the above two constraints becomes an optimization problem over the intersection of the oblique manifold and the Stiefel manifold. We propose to use the steepest descent method on the intersection of these two manifolds to obtain the optimal solution. By using the alternating projections method, the search direction of the steepest descent method is derived. Meanwhile, the convergence analysis for the proposed approach is also provided. The proposed approach can be used for both omnidirectional and sector-shaped power pattern design. Simulation results show that the power pattern of our designed precoder has less variation in different spatial directions within the cell coverage compared with the existing Zadoff-Chu scheme.

In this paper, we design the precoding matrix with broad coverage for massive multi-input multi-output (MIMO) public channels. To sufficiently utilize the power amplifier (PA) capacity of the base station (BS), all the rows of the precoding matrix should have the same 2-norm to guarantee equal transmit power on each antenna. In the meanwhile, the precoding matrix is semi-unitary, which can maximize the achievable ergodic rate for the independent and identically distributed (i.i.d.) channel. Within the framework of manifold optimization, the precoding matrix design under the above two constraints becomes an optimization problem over the intersection of the oblique manifold and the Stiefel manifold. We propose to use the steepest descent method on the intersection of these two manifolds to obtain the optimal solution. By using the alternating projections method, the search direction of the steepest descent method is derived. In the meanwhile, the convergence analysis for the proposed approach is also provided. Simulation results show that the power pattern of our designed precoder has less variation in different spatial directions within the cell compared with the existing schemes.