Distributed Mean Estimation (DME), in which $n$ clients communicate vectors to a parameter server that estimates their average, is a fundamental building block in communication-efficient federated learning. In this paper, we improve on previous DME techniques that achieve the optimal $O(1/n)$ Normalized Mean Squared Error (NMSE) guarantee by asymptotically improving the complexity for either encoding or decoding (or both). To achieve this, we formalize the problem in a novel way that allows us to use off-the-shelf mathematical solvers to design the quantization. Using various datasets and training tasks, we demonstrate how QUIC-FL achieves state of the art accuracy with faster encoding and decoding times compared to other DME methods.

BERT based ranking models have achieved superior performance on various information retrieval tasks.However, the large number of parameters and complex self-attention operations come at a significant latency overhead.To remedy this, recent works propose late-interaction architectures, which allow precomputation of intermediate document representations, thus reducing latency.Nonetheless, having solved the immediate latency issue, these methods now introduce storage costs and network fetching latency, which limit their adoption in real-life production systems.In this work, we propose the Succinct Document Representation (SDR) scheme that computes highly compressed intermediate document representations, mitigating the storage/network issue.Our approach first reduces the dimension of token representations by encoding them using a novel autoencoder architecture that uses the document's textual content in both the encoding and decoding phases.After this token encoding step, we further reduce the size of the document representations using modern quantization techniques.Evaluation on MSMARCO's passage rereranking task show that compared to existing approaches using compressed document representations, our method is highly efficient, achieving 4x-11.6xhigher compression rates for the same ranking quality.Similarly, on the TREC CAR dataset, we achieve 7.7x higher compression rate for the same ranking quality.

Distributed Mean Estimation (DME) is a central building block in federated learning, where clients send local gradients to a parameter server for averaging and updating the model. Due to communication constraints, clients often use lossy compression techniques to compress the gradients, resulting in estimation inaccuracies. DME is more challenging when clients have diverse network conditions, such as constrained communication budgets and packet losses. In such settings, DME techniques often incur a significant increase in the estimation error leading to degraded learning performance. In this work, we propose a robust DME technique named EDEN that naturally handles heterogeneous communication budgets and packet losses. We derive appealing theoretical guarantees for EDEN and evaluate it empirically. Our results demonstrate that EDEN consistently improves over state-of-the-art DME techniques.

We consider the problem where $n$ clients transmit $d$-dimensional real-valued vectors using $d(1+o(1))$ bits each, in a manner that allows the receiver to approximately reconstruct their mean. Such compression problems naturally arise in distributed and federated learning. We provide novel mathematical results and derive computationally efficient algorithms that are more accurate than previous compression techniques. We evaluate our methods on a collection of distributed and federated learning tasks, using a variety of datasets, and show a consistent improvement over the state of the art.

Summary This paper describes TRUSTPACK, a decentralized trust management framework that provides trust management as a generic service. TRUSTPACK is unique in that it does not provide a central service. Instead, it is run by many autonomous services. This design enables TRUSTPACK to alleviate privacy concerns, as well as potentially provide better personalization and scalability when compared with current centralized solutions. A major component of TRUSTPACK is a generic decentralized graph query processing framework called GRAPHPACK, which was also developed as part of this work. GRAPHPACK consists of a decentralized graph processing language as well as an execution engine, as elaborated in this paper. The paper also presents several examples and a case study showing how TRUSTPACK can be used to handle various trust management scenarios, as well as its incorporation in an existing third party P2P file sharing application. Prototypes of TRUSTPACK and GRAPHPACK are available as open source projects at http://code.google.com/p/trustpack/ and http://code.google.com/p/graphpack/ , respectively. Copyright © 2013 John Wiley & Sons, Ltd.