Regret analysis is challenging in Multi-Agent Reinforcement Learning (MARL) primarily due to the dynamical environments and the decentralized information among agents. We attempt to solve this challenge in the context of decentralized learning in multi-agent linear-quadratic (LQ) dynamical systems. We begin with a simple setup consisting of two agents and two dynamically decoupled stochastic linear systems, each system controlled by an agent. The systems are coupled through a quadratic cost function. When both systems’ dynamics are unknown and there is no communication among the agents, we show that no learning policy can generate sub-linear in $T$ regret, where $T$ is the time horizon. When only one system’s dynamics are unknown and there is one-directional communication from the agent controlling the unknown system to the other agent, we propose a MARL algorithm based on the construction of an auxiliary single-agent LQ problem. The auxiliary single-agent problem in the proposed MARL algorithm serves as an implicit coordination mechanism among the two learning agents. This allows the agents to achieve a regret within $O(\sqrt{T})$ of the regret of the auxiliary single-agent problem. Consequently, using existing results for single-agent LQ regret, our algorithm provides a $\tilde{O}(\sqrt{T})$ regret bound. (Here $\tilde{O}(\cdot)$ hides constants and logarithmic factors). Our numerical experiments indicate that this bound is matched in practice. From the two-agent problem, we extend our results to multi-agent LQ systems with certain communication patterns which appear in vehicle platoon control.

Coded caching can significantly reduce the communication bandwidth requirement for satisfying users' demands by utilizing the multicasting gain among multiple users. Most existing works assume that the users follow the prescriptions for content placement made by the system. However, users may prefer to decide what files to cache. To address this issue, we consider a network consisting of a file server connected through a shared link to K users, each equipped with a cache which has been already filled arbitrarily. Given an arbitrary content placement, the goal is to find a delivery strategy for the server that minimizes the load of the shared link. In this paper, we focus on a specific class of coded multicasting delivery schemes known as the “clique cover delivery scheme.” We first formulate the optimal clique cover delivery problem as a combinatorial optimization problem. Using a connection with the weighted set cover problem, we propose an approximation algorithm and show that it provides an approximation ratio of (1 + log K), while the approximation ratio for the existing coded delivery schemes is linear in K. Numerical simulations show that our proposed algorithm provides a considerable bandwidth reduction over the existing coded delivery schemes for almost all content placement schemes.

A new class of caching schemes, called coded caching, can significantly reduce the communication bandwidth requirement for satisfying users' demands by utilizing the multicasting gain among multiple users. Most existing works assume that the users follow the prescriptions for content placement made by the system. However, users may prefer to decide what files to cache or discard some part of their caches due to lack of space. In order to address this issue, we study a caching system where the content placement phase has been already carried out by the users arbitrarily. More specifically, we consider a network consisting of a file server connected through a shared link to $K$ users, each equipped with a cache. Given arbitrary content placement by the users, the goal is to find a coded multicast strategy for the server that minimizes the load of the shared link. We first formulate the optimal coded multicast design problem as an Integer Linear Program (ILP). Using a connection with the weighted set cover problem, we propose an approximation algorithm for solving this problem. We show that our proposed algorithm provides $(1 + \log K)$-approximation for the optimal coded multicast design problem, while the approximation ratio for the existing coded delivery schemes is linear in $K$. Numerical simulations show that our proposed algorithm provides a considerable bandwidth reduction over the existing coded delivery schemes for uniformly-random generated content placement.

We study the set of continuous functions that admit no spurious local optima (i.e. local minima that are not global minima) which we term global functions. They satisfy various powerful properties for analyzing nonconvex and nonsmooth optimization problems. For instance, they satisfy a theorem akin to the fundamental uniform limit theorem in the analysis regarding continuous functions. Global functions are also endowed with useful properties regarding the composition of functions and change of variables. Using these new results, we show that a class of non-differentiable nonconvex optimization problems arising in tensor decomposition applications are global functions. This is the first result concerning nonconvex methods for nonsmooth objective functions. Our result provides a theoretical guarantee for the widely-used $\ell_1$ norm to avoid outliers in nonconvex optimization.

We consider the problem of learning an unknown Markov Decision Process (MDP) that is weakly communicating in the infinite horizon setting. We propose a Thompson Sampling-based reinforcement learning algorithm with dynamic episodes (TSDE). At the beginning of each episode, the algorithm generates a sample from the posterior distribution over the unknown model parameters. It then follows the optimal stationary policy for the sampled model for the rest of the episode. The duration of each episode is dynamically determined by two stopping criteria. The first stopping criterion controls the growth rate of episode length. The second stopping criterion happens when the number of visits to any state-action pair is doubled. We establish $\tilde O(HS\sqrt{AT})$ bounds on expected regret under a Bayesian setting, where $S$ and $A$ are the sizes of the state and action spaces, $T$ is time, and $H$ is the bound of the span. This regret bound matches the best available bound for weakly communicating MDPs. Numerical results show it to perform better than existing algorithms for infinite horizon MDPs.

In this paper an optimal relay selection algorithm with non-negligible probing time is proposed and analyzed for cooperative wireless networks. Relay selection has been introduced to solve the degraded bandwidth efficiency problem in cooperative communication. Yet complete information of relay channels often remain unavailable for complex networks which renders the optimal selection strategies impossible for transmission source without probing the relay channels. Particularly when the number of relay candidate is large, even though probing all relay channels guarantees the finding of the best relays at any time instant, the degradation of bandwidth efficiency due to non-negligible probing times, which was often neglected in past literature, is also significant. In this work, a stopping rule based relay selection strategy is determined for the source node to decide when to stop the probing process and choose one of the probed relays to cooperate with under wireless channels' stochastic uncertainties. This relay selection strategy is further shown to have a simple threshold structure. At the meantime, full diversity order and high bandwidth efficiency can be achieved simultaneously. Both analytical and simulation results are provided to verify the claims.

We consider a multiple access communication system where multiple users share a common collision channel. In this system, coordination among users is essential to resolve collision issues. However, each system user can only observe its own local traffic and the feedback from the channel. Without a centralized controller, it is challenging to design an efficient coordination protocol. We present a decentralized common information-based multiple access protocol (CIMA). We show that under CIMA collision is totally avoided without channel sensing and the full throughput region of the collision channel is achieved. In addition, simulation results indicate that the CIMA protocol incurs low transmission delay.

We consider the channel sensing problem arising in opportunistic scheduling over fading channels, cognitive radio networks, and resource constrained jamming. The same problem arises in many other areas of science and technology as it is an instance of restless bandit problems. The communication system consists of N channels. Each channel is modeled as a multi-state Markov chain. At each time instant a user selects one channel to sense and uses it to transmit information. A reward depending on the state of the selected channel is obtained for each transmission. The objective is to design a channel sensing policy that maximizes the expected total reward collected over a finite or infinite horizon. This problem can be viewed as an instance of restless bandit problems, for which the form of optimal policies is unknown in general. We discover sets of conditions sufficient to guarantee the optimality of a myopic sensing policy; we show that under one particular set of conditions the myopic policy coincides with the Gittins index rule.