We consider correlated equilibria in an adversarial environment, where an adversary can compromise the public signal used by the players for choosing their strategies, while players aim at detecting a potential attack as soon as possible to avoid loss of utility. We model the interaction between the adversary and the players as a zero-sum game and we derive the maxmin strategies for both the defender and the attacker using the framework of quickest change detection. We define a class of adversarial strategies that achieve the optimal trade-off between the impact and the detectability of the attack for the adversary and show that a generalized CUSUM scheme is asymptotically optimal for their detection. Our numerical results on the Sioux-Falls benchmark traffic routing game show that the proposed detection scheme can effectively limit the utility loss by a potential adversary.

We address the problem of detecting adversarial attacks against cooperative multi-agent reinforcement learning with continuous action space. We propose a decentralized detector that relies solely on the local observations of the agents and makes use of a statistical characterization of the normal behavior of observable agents. The proposed detector utilizes deep neural networks to approximate the normal behavior of agents as parametric multivariate Gaussian distributions. Based on the predicted density functions, we define a normality score and provide a characterization of its mean and variance. This characterization allows us to employ a two-sided CUSUM procedure for detecting deviations of the normality score from its mean, serving as a detector of anomalous behavior in real-time. We evaluate our scheme on various multi-agent PettingZoo benchmarks against different state-of-the-art attack methods, and our results demonstrate the effectiveness of our method in detecting impactful adversarial attacks. Particularly, it outperforms the discrete counterpart by achieving AUC-ROC scores of over 0.95 against the most impactful attacks in all evaluated environments.

In this paper, we propose a cache update policy for wireless networks considering dynamic popularity for file requests. Our network scenario is a wireless network comprised of some cache-equipped access points (APs) which are deployed densely in an area and have connections with core network servers. We model the dynamics of file requests as Markov-modulated Poisson processes. Considering the congestion-dependent delay of APs' services and their overlapping coverage areas, we model the cache service for the cache-missed requests, as a stochastic game in which a Vickrey–Clarke–Groves (VCG) mechanism is exploited at each stage of the game as our proposed file routing policy, and the APs decide independently how to update their caches regarding the routed files. To this end, the APs' utilities are defined based on long-term average file delivery delay including the queueing delay in APs. By finding the Nash equilibrium of the game, we propose policies for both delivery and cache update of dynamic contents. Finally, by comparing numerical results of our proposed scheme with some conventional caching schemes, we show significant improvements in network performance in terms of file delivery delay in different conditions.

We consider the problem of detecting adversarial attacks against cooperative multi-agent reinforcement learning. We propose a decentralized scheme that allows agents to detect the abnormal behavior of one compromised agent. Our approach is based on a recurrent neural network (RNN) trained during cooperative learning to predict the action distribution of other agents based on local observations. The predicted distribution is used for computing a normality score for the agents, which allows the detection of the misbehavior of other agents. To explore the robustness of the proposed detection scheme, we formulate the worst-case attack against our scheme as a constrained reinforcement learning problem. We propose to compute an attack policy by optimizing the corresponding dual function using reinforcement learning. Extensive simulations on various multi-agent benchmarks show the effectiveness of the proposed detection scheme in detecting state-of-the-art attacks and in limiting the impact of undetectable attacks.