In real-world networks, we regularly face the effect of propagating failures over networks, for example, rumors spread over social networks, outages spread over power networks, viruses spread over communication and biological networks. Often, these failures spread over a network of agents with unknown and potentially diverse degrees of vulnerabilities to the propagating phenomenon. In this work, we consider a general network model subject to propagating failures and develop provably fast mechanisms for learning the unknown vulnerabilities of the network with minimal cost incurred in the process. We propose an extension to the classic Independent Cascade (IC) model where we incorporate both node and edge failures with non-uniform costs. From an online learning perspective, the goal is to find an optimal policy to control where to start failures and generate samples. Therefore, we formulate a cost minimization problem with Probably-Approximately-Correct (PAC) type guarantees. As a theoretical benchmark, we design a linear programming problem using a proposed joint Bernstein inequality. Then we characterize the performance of randomized policies that use a fixed budget distribution independent of sampling history. Finally, we propose a fast Lyapunov-based online learning policy, for which we give a formal theoretical analysis. The performance of the policy are validated under extensive numerical studies for both synthetic and real-world networks.

Mean-field games have been used as a theoretical tool to obtain an approximate Nash equilibrium for symmetric and anonymous $N$-player games. However, limiting applicability, existing theoretical results assume variations of a “population generative model”, which allows arbitrary modifications of the population distribution by the learning algorithm. Moreover, learning algorithms typically work on abstract simulators with population instead of the $N$-player game. Instead, we show that $N$ agents running policy mirror ascent converge to the Nash equilibrium of the regularized game within $\widetilde{\mathcal{O}}(\varepsilon^{-2})$ samples from a single sample trajectory without a population generative model, up to a standard $\mathcal{O}(\frac{1}{\sqrt{N}})$ error due to the mean field. Taking a divergent approach from the literature, instead of working with the best-response map we first show that a policy mirror ascent map can be used to construct a contractive operator having the Nash equilibrium as its fixed point. We analyze single-path TD learning for $N$-agent games, proving sample complexity guarantees by only using a sample path from the $N$-agent simulator without a population generative model. Furthermore, we demonstrate that our methodology allows for independent learning by $N$ agents with finite sample guarantees.

In a broad class of reinforcement learning applications, stochastic rewards have heavy-tailed distributions, which lead to infinite second-order moments for stochastic (semi)gradients in policy evaluation and direct policy optimization. In such instances, the existing RL methods may fail miserably due to frequent statistical outliers. In this work, we establish that temporal difference (TD) learning with a dynamic gradient clipping mechanism, and correspondingly operated natural actor-critic (NAC), can be provably robustified against heavy-tailed reward distributions. It is shown in the framework of linear function approximation that a favorable tradeoff between bias and variability of the stochastic gradients can be achieved with this dynamic gradient clipping mechanism. In particular, we prove that robust versions of TD learning achieve sample complexities of order $\mathcal{O}(\varepsilon^{-\frac{1}{p}})$ and $\mathcal{O}(\varepsilon^{-1-\frac{1}{p}})$ with and without the full-rank assumption on the feature matrix, respectively, under heavy-tailed rewards with finite moments of order $(1+p)$ for some $p\in(0,1]$, both in expectation and with high probability. We show that a robust variant of NAC based on Robust TD learning achieves $\tilde{\mathcal{O}}(\varepsilon^{-4-\frac{2}{p}})$ sample complexity. We corroborate our theoretical results with numerical experiments.

In a wide variety of applications including online advertising, contractual hiring, and wireless scheduling, the controller is constrained by a stringent budget constraint on the available resources, which are consumed in a random amount by each action, and a stochastic feasibility constraint that may impose important operational limitations on decision-making. In this work, we consider a general model to address such problems, where each action returns a random reward, cost, and penalty from an unknown joint distribution, and the decision-maker aims to maximize the total reward under a budget constraint B on the total cost and a stochastic constraint on the time-average penalty. We propose a novel low-complexity algorithm based on Lyapunov optimization methodology, named LyOn, and prove that for K arms it achieves square root of KBlog(B) regret and zero constraint-violation when B is sufficiently large. The low computational cost and sharp performance bounds of LyOn suggest that Lyapunov-based algorithm design methodology can be effective in solving constrained bandit optimization problems.

We consider a budget-constrained bandit problem where each arm pull incurs a random cost, and yields a random reward in return. The objective is to maximize the total expected reward under a budget constraint on the total cost. The model is general in the sense that it allows correlated and potentially heavy-tailed cost-reward pairs that can take on negative values as required by many applications. We show that if moments of order $(2+\gamma)$ for some $\gamma > 0$ exist for all cost-reward pairs, $O(\log B)$ regret is achievable for a budget $B>0$. In order to achieve tight regret bounds, we propose algorithms that exploit the correlation between the cost and reward of each arm by extracting the common information via linear minimum mean-square error estimation. We prove a regret lower bound for this problem, and show that the proposed algorithms achieve tight problem-dependent regret bounds, which are optimal up to a universal constant factor in the case of jointly Gaussian cost and reward pairs.

The theory of discrete-time online learning has been successfully applied in many problems that involve sequential decision-making under uncertainty. However, in many applications including contractual hiring in online freelancing platforms and server allocation in cloud computing systems, the outcome of each action is observed only after a random and action-dependent time. Furthermore, as a consequence of certain ethical and economic concerns, the controller may impose deadlines on the completion of each task, and require fairness across different groups in the allocation of total time budget $B$. In order to address these applications, we consider continuous-time online learning problem with fairness considerations, and present a novel framework based on continuous-time utility maximization. We show that this formulation recovers reward-maximizing, max-min fair and proportionally fair allocation rules across different groups as special cases. We characterize the optimal offline policy, which allocates the total time between different actions in an optimally fair way (as defined by the utility function), and impose deadlines to maximize time-efficiency. In the absence of any statistical knowledge, we propose a novel online learning algorithm based on dual ascent optimization for time averages, and prove that it achieves $\tilde{O}(B^{-1/2})$ regret bound.

We study the problem of serving randomly arriving and delay-sensitive traffic over a multi-channel communication system with time-varying channel states and unknown statistics. This problem deviates from the classical exploration-exploitation setting in that the design and analysis must accommodate the dynamics of packet availability and urgency as well as the cost of each channel use at the time of decision. To that end, we have developed and investigated an index-based policy upper confidence bound (UCB)-deadline, which performs dynamic channel allocation decisions that incorporate these traffic requirements and costs. Under symmetric channel conditions, we have proved that the UCB-deadline policy can achieve bounded regret in the likely case where the cost of using a channel is not too high to prevent all transmissions, and logarithmic regret otherwise. In this case, we show that UCB-deadline is order-optimal. We also perform numerical investigations to validate the theoretical fundings, and also compare the performance of the UCB-deadline to another learning algorithm that we propose based on Thompson sampling.

We study the problem of serving randomly arriving and delay-sensitive traffic over a multi-channel communication system with time-varying channel states and unknown statistics. This problem deviates from the classical exploration-exploitation setting in that the design and analysis must accommodate the dynamics of packet availability and urgency as well as the cost of each channel use at the time of decision. To that end, we have developed and investigated two policies, one index-based (UCB-Deadline) and the other Bayesian (TS-Deadline), both of which perform dynamic channel allocation decisions that incorporate these traffic requirements and costs. Under symmetric channel conditions, we have proved that the UCB-Deadline policy can achieve bounded regret in the likely case where the cost of using a channel is not too high to prevent all transmissions, and logarithmic regret otherwise. In our numerical studies, we also show that TS-Deadline achieves superior performance over its UCB counterpart, making it a potentially useful alternative when fast convergence to optimal is important.

Advances in millimeter-wave (mmW) communication open up a vast frequency band, from 30 - 300 GHz, for use in mobile communication systems. However, these new frequencies exhibit highly variable and intermittent channel characteristics. In this work, we investigate the impact of diverse mmW channel statistics on the limit and the rate at which the achievable rates converge to the infinite bandwidth capacity. In particular, we first identify the optimal power allocation algorithm, and investigate the behavior of the capacity both asymptotically and at finite number of channels. Then, we propose a suboptimal algorithm that achieves asymptotic optimality and is tractable for analysis, and analyze the convergence rate of the achievable rate under this algorithm. Analytical findings are supported by numerical investigations in realistic communication scenarios.

In this paper, lossless polar compression of q-ary memoryless sources in the noiseless setting is investigated. Polar compression scheme for binary memoryless sources, introduced by Cronie and Korada, is generalized to sources over prime-size alphabets. In order to reduce the average codeword length, a compression scheme based on successive cancellation list decoding is proposed. Also, a specific configuration for the compression of correlated sources is considered, and it is shown that the introduced polar compression schemes achieve the corner point of the admissible rate region. Based on this result, proposed compression schemes are extended to arbitrary finite source alphabets by using a layered approach.