The high communication cost between the server and the clients is a significant bottleneck in scaling distributed learning for overparametrized deep models. One popular approach for reducing this communication overhead is randomized sketching. However, existing theoretical analyses for sketching-based distributed learning (sketch-DL) either incur a prohibitive dependence on the ambient dimension or need additional restrictive assumptions such as heavy-hitters. Nevertheless, despite existing pessimistic analyses, empirical evidence suggests that sketch-DL is competitive with its uncompressed counterpart, thus motivating a sharper analysis. In this work, we introduce a sharper ambient dimension-independent convergence analysis for sketch-DL using the second-order geometry specified by the loss Hessian. Our results imply ambient dimension-independent communication complexity for sketch-DL. We present empirical results both on the loss Hessian and overall accuracy of sketch-DL supporting our theoretical results. Taken together, our results provide theoretical justification for the observed empirical success of sketch-DL.

In this paper, we consider a bandit problem in which there are a number of groups each consisting of infinitely many arms. Whenever a new arm is requested from a given group, its mean reward is drawn from an unknown reservoir distribution (different for each group), and the uncertainty in the arm’s mean reward can only be reduced via subsequent pulls of the arm. The goal is to identify the infinite-arm group whose reservoir distribution has the highest $(1-\alpha)$-quantile (e.g., median if $\alpha = \frac{1}{2}$), using as few total arm pulls as possible. We introduce a two-step algorithm that first requests a fixed number of arms from each group and then runs a finite-arm grouped max-quantile bandit algorithm. We characterize both the instance-dependent and worst-case regret, and provide a matching lower bound for the latter, while discussing various strengths, weaknesses, algorithmic improvements, and potential lower bounds associated with our instance-dependent upper bounds.

Temporal graph representation learning (T-GRL) aims to learn representations that model how graph edges evolve over time. While recent works on T-GRL have improved link prediction accuracy in temporal settings, their methods optimize a point-wise loss function independently over future links rather than optimize jointly over a candidate set per node. In applications where resources (e.g., attention) are allocated based on ranking links by likelihood, the use of a ranking loss is preferred. However it is not straightforward to develop a T-GRL method to optimize a ranking loss due to a tradeoff between model expressivity and scalability. In this work, we address these issues and propose a Temporal Graph network for Ranking (TGRank), which significantly improves performance for link prediction tasks by (i) optimizing a list-wise loss for improved ranking, and (ii) incorporating a labeling approach designed to allow for efficient inference over the candidate set jointly, while provably boosting expressivity. We extensively evaluate TGRank over six real networks. TGRank outperforms the state-of-the-art baselines on average by 14.21%↑ (transductive) and 16.25% ↑ (inductive) in ranking metrics while being more efficient (up-to 65 × speed-up) to make inference on large networks.

Modern users demand analytical features on fresh, real time data. Offering these analytical features to hundreds of millions of users is a relevant problem encountered by many large scale web companies.

The complexity and continuous evolution of enterprise systems is making it increasingly difficult to maintain the predictability of the system. The system managers are often unaware of the impact of an action across various layers of enterprise IT ranging from business functions, to applications, to IT infrastructure. Due to lack of this transparency, many risks go unnoticed leading to business outages or violations of Service Level Agreements (SLAs). In this paper, we address this problem of impact analysis of an action of an IT component on the enterprise system. We present novel techniques to model various aspects of system dependencies and then present algorithms to perform impact analysis. The ideas presented in this paper have been developed by applying them in real-world environments.