We revisit the problem of designing scalable protocols for private statistics and private federated learning when each device holds its private data. Locally differentially private algorithms require little trust but are (provably) limited in their utility. Centrally differentially private algorithms can allow significantly better utility but require a trusted curator. This gap has led to significant interest in the design and implementation of simple cryptographic primitives, that can allow central-like utility guarantees without having to trust a central server.

We study adaptive methods for differentially private convex optimization, proposing and analyzing differentially private variants of a Stochastic Gradient Descent (SGD) algorithm with adaptive stepsizes, as well as the AdaGrad algorithm. We provide upper bounds on the regret of both algorithms and show that the bounds are (worst-case) optimal. As a consequence of our development, we show that our private versions of AdaGrad outperform adaptive SGD, which in turn outperforms traditional SGD in scenarios with non-isotropic gradients where (non-private) Adagrad provably outperforms SGD. The major challenge is that the isotropic noise typically added for privacy dominates the signal in gradient geometry for high-dimensional problems; approaches to this that effectively optimize over lower-dimensional subspaces simply ignore the actual problems that varying gradient geometries introduce. In contrast, we study non-isotropic clipping and noise addition, developing a principled theoretical approach; the consequent procedures also enjoy significantly stronger empirical performance than prior approaches.

Anonymous systems on the Internet aim to protect users from revealing to an external unauthorized entity their identities and their network activities. Despite using layered encryption, these systems are still vulnerable to timing analysis, wherein an eavesdropper can use traffic correlation mechanisms to identify the source of packets arriving at a destination. Mixes are intelligent routers or proxy servers that aim to provide packet source anonymity from timing analysis by delaying and shuffling the order of received packets prior to transmission. Such shuffling strategies naturally increase latency and result in a tradeoff between anonymity and latency. This paper investigates this tradeoff in a network of mixes, by deriving the optimal routing for sources which maximizes weighted sum of anonymity and delay. The achievable anonymity is characterized analytically for a general multipath model, and it is shown that under light traffic conditions, there exists a unique single route strategy, which achieves the optimal delay anonymity tradeoff. A low complexity algorithm is presented that derives the optimal routes to achieve a desired tradeoff. The light traffic results are specialized for a graphical model of existing practical anonymous systems, and optimal scaling behavior with the size of such networks is characterized. In the heavy traffic regime, it is shown that optimal anonymity is achieved for any allocation of rates across the different routes. Simulations on example networks are presented where it is shown that the optimal routes derived under light traffic performs quite well in general traffic regime.