Selecting suitable architecture parameters and training hyperparameters is essential for enhancing machine learning (ML) model performance. Several recent empirical studies conduct large-scale correlational analysis on neural networks (NNs) to search for effective generalization metrics that can guide this type of model selection. Effective metrics are typically expected to correlate strongly with test performance. In this paper, we expand on prior analyses by examining generalization-metric-based model selection with the following objectives: (i) focusing on natural language processing (NLP) tasks, as prior work primarily concentrates on computer vision (CV) tasks; (ii) considering metrics that directly predict test error instead of the generalization gap; (iii) exploring metrics that do not need access to data to compute. From these objectives, we are able to provide the first model selection results on large pretrained Transformers from Huggingface using generalization metrics. Our analyses consider (I) hundreds of Transformers trained in different settings, in which we systematically vary the amount of data, the model size and the optimization hyperparameters, (II) a total of 51 pretrained Transformers from eight families of Huggingface NLP models, including GPT2, BERT, etc., and (III) a total of 28 existing and novel generalization metrics. Despite their niche status, we find that metrics derived from the heavy-tail (HT) perspective are particularly useful in NLP tasks, exhibiting stronger correlations than other, more popular metrics. To further examine these metrics, we extend prior formulations relying on power law (PL) spectral distributions to exponential (EXP) and exponentially-truncated power law (E-TPL) families.

Federated learning (FL) is a promising way to use the computing power of mobile devices while maintaining the privacy of users. Current work in FL, however, makes the unrealistic assumption that the users have ground-truth labels on their devices, while also assuming that the server has neither data nor labels. In this work, we consider the more realistic scenario where the users have only unlabeled data, while the server has some labeled data, and where the amount of labeled data is smaller than the amount of unlabeled data. We call this learning problem semi-supervised federated learning (SSFL). For SSFL, we demonstrate that a critical issue that affects the test accuracy is the large gradient diversity of the models from different users. Based on this, we investigate several design choices. First, we find that the so-called consistency regularization loss (CRL), which is widely used in semi-supervised learning, performs reasonably well but has large gradient diversity. Second, we find that Batch Normalization (BN) increases gradient diversity. Replacing BN with the recently-proposed Group Normalization (GN) can reduce gradient diversity and improve test accuracy. Third, we show that CRL combined with GN still has a large gradient diversity when the number of users is large. Based on these results, we propose a novel grouping-based model averaging method to replace the FedAvg averaging method. Overall, our grouping-based averaging, combined with GN and CRL, achieves better test accuracy than not just a contemporary paper on SSFL in the same settings (>10%), but also four supervised FL algorithms.

In this paper, we consider hybrid parallelism---a paradigm that employs both Data Parallelism (DP) and Model Parallelism (MP)---to scale distributed training of large recommendation models. We propose a compression framework called Dynamic Communication Thresholding (DCT) for communication-efficient hybrid training. DCT filters the entities to be communicated across the network through a simple hard-thresholding function, allowing only the most relevant information to pass through. For communication efficient DP, DCT compresses the parameter gradients sent to the parameter server during model synchronization. The threshold is updated only once every few thousand iterations to reduce the computational overhead of compression. For communication efficient MP, DCT incorporates a novel technique to compress the activations and gradients sent across the network during the forward and backward propagation, respectively. This is done by identifying and updating only the most relevant neurons of the neural network for each training sample in the data. We evaluate DCT on publicly available natural language processing and recommender models and datasets, as well as recommendation systems used in production at Facebook. DCT reduces communication by at least 100x and 20x during DP and MP, respectively. The algorithm has been deployed in production, and it improves end-to-end training time for a state-of-the-art industrial recommender model by 37%, without any loss in performance.

Motivated by recent developments in serverless systems for large-scale computation as well as improvements in scalable randomized matrix algorithms, we develop OverSketched Newton, a randomized Hessian-based optimization algorithm to solve large-scale convex optimization problems in serverless systems. OverSketched Newton leverages matrix sketching ideas from Randomized Numerical Linear Algebra to compute the Hessian approximately. These sketching methods lead to inbuilt resiliency against stragglers that are a characteristic of serverless architectures. Depending on whether or not the problem is strongly convex, we propose different iteration updates using the approximate Hessian. For both cases, we establish convergence guarantees for OverSketched Newton, and we empirically validate our results by solving large-scale supervised learning problems on real-world datasets. Experiments demonstrate a reduction of ~50% in total running time on AWS Lambda, compared to state-of-the-art distributed optimization schemes.

We propose OverSketch, an approximate algorithm for distributed matrix multiplication in serverless computing. OverSketch leverages ideas from matrix sketching and high-performance computing to enable cost-efficient multiplication that is resilient to faults and straggling nodes pervasive in low-cost serverless architectures. We establish statistical guarantees on the accuracy of OverSketch and empirically validate our results by solving a large-scale linear program using interior-point methods and demonstrate a 34% reduction in compute time on AWS Lambda.

The ability to search databases by providing multimedia examples of voices, faces, or locations instead of textual descriptions can be tremendously useful. At the same time, uploading media for queries---especially media that contains sensitive content---means sharing private information with a potentially untrusted service provider. The growing field of privacy-preserving database searches attempts to resolve this tension. Within this scope of private searches, private media classification and retrieval is particularly challenging due to the inherent inexactness of recognition; to be useful, image or other media classification systems must identify approximate matches rather than just exact ones. This is difficult to reconcile with distortion-intolerant and resource-heavy privacy primitives, especially in web-scale databases. In this paper, we present an architecture for media classification on public databases that preserves client privacy while achieving asymptotic communication and computation costs that are sublinear in the size of the database. We demonstrate the usefulness of this architecture in the context of a privacy-preserving face recognition system. We observe order-of-magnitude speedups over state-of-the-art private face recognition systems.

We consider video sequences that have been encrypted uncompressed. Since encryption masks the source, traditional data compression algorithms are rendered ineffective. However, it has been shown that through the use of distributed source-coding techniques, the compression of encrypted data is in fact possible. This means that it is possible to reduce data size without requiring that the data be compressed prior to encryption. Indeed, under some reasonable conditions, neither security nor compression efficiency need be sacrificed when compression is performed on the encrypted data (Johnson et al., 2004). In this paper we develop an algorithm for the practical lossless compression of encrypted gray scale video. Our method is based on considering the temporal correlations in the video. This move to temporal dependence builds on our previous work on memoryless sources, and one- and two-dimensional Markov sources. For comparison, a motion-compensated lossless video encoder can compress each unencrypted frame of the standard "Foreman" test video sequence by about 57%. Our algorithm can compress the same frames, after encryption, by about 33%

Slepian-Wolf coding tackles the problem of distributed encoding of correlated discrete-alphabet sources for decoding at a common receiver. In this work, we propose a distributed linear block code construction for attaining any point on the Slepian-Wolf achievable rate region for arbitrarily correlated sources using only a single code. Specifically, our prescription allows for any arbitrary memoryless joint probability distribution over any arbitrary number of distributed sources, and allows for any arbitrary rate combination that lies in the Slepian-Wolf achievable region. Special cases of our framework include the single source case (wherein our construction reduces to an entropy coder), source coding with side-information at the receiver (so-called corner points of the Slepian-Wolf region), and specific source correlation models (such as induced by a virtual Binary Symmetric Channel model). In this work, we describe how to use low density parity check (LDPC) codes in the proposed framework to solve the general Slepian-Wolf problem constructively.

The problem of rate-distortion efficient constructions is studied for the problem of source coding with side information (SCSI), which has assumed heightened interest. While the Wyner-Ziv theorem from information theory has prescribed rate-distortion performance bounds for the SCSI problem, the gap between theory and practice has remained large. To reduce this gap, two different frameworks are proposed based on a trellis construction and a turbo-based construction respectively. Simulation results on the Gaussian SCSI problem reveal the promise of the proposed approaches: at 1 bit per sample, 0.5 bits/sample, 0.25 bits/sample and 0.125 bits/sample, these constructions attain performance within 1.3 dB, 1.1 dB, 0.85 dB and 0.5 dB respectively of the theoretical Wyner-Ziv rate-distortion bound.

We present new achievable rate regions and code constructions for the symmetric n-channel multiple descriptions (MD) coding problem (Puri et al. (2002)) for n>2. Our approach is inspired by unexplored connections between MD and the problem of distributed source coding (Slepian et al. (1973); Wyner et al. (1976)). For illustrative clarity, we restrict our focus to the important special case relating to (n, k) source-channel erasure codes (Pradhan et al. (2001)). This involves n encodings of a source with the goal of maximizing its reconstruction fidelity with the availability of any k of them, while strictly improving this reconstruction fidelity with the availability of more than k descriptions. We describe the underlying information-theoretic framework, and then formulate practical constructions based on scalar quantizers and linear channel codes for the n=3 case to illustrate our concepts.

We address digital transmission for enhancing, in a backward compatible way, the quality of analog image transmission systems. We propose a practical algorithm that treats the problem as one of wavelet image compression with side information (available in the form of a noisy analog version of the image) present at the decoder. We propose a rate allocation technique to efficiently allocate the rate among the wavelet coefficients of the image. In typical instances of the problem, we get gain up to 2.5 dB over conventional methods that ignore the side information. Surprisingly, this is typically achieved by modifying a very small fraction of the wavelet coefficients (typically around 10-20%) of the conventional source coder. Extensions of our proposed image transmission framework to that of video transmission finds application in the upgrade of current analog television broadcast systems to digital TV.

We address the problem of distributed source coding using a practical and constructive approach, referred to as distributed source coding using syndromes (DISCUS), with applications to sensor networks. We propose low complexity encoding and decoding methods based on linear codes, to achieve all points in the achievable rate region of the Slepian-Wolf (1973) problem. The extension of these concepts to the construction of Euclidean-space codes is also studied and analyzed for the case of trellis and lattice codes. The performance of these symmetric methods for encoding with a fidelity criterion is shown to be the same as that of asymmetric encoding. Simulations are presented to corroborate these results.

Summary form only given. This work introduces a multiple-description product code which aims at optimally generating multiple, equally-important wavelet image descriptions from an image encoded by the popular SPIHT image coder. Because the SPIHT image coder is highly sensitive to errors, forward error correction is used to protect the image against bit errors occurring in the channel. The error-correction code is a concatenated channel code including a row (outer) code based on RCPC codes with CRC error detection and a source-channel column (inner) code consisting of the scalable SPIHT image coder and an optimized array of unequal protection Reed-Solomon erasure-correction codes. By matching the unequal protection codes to the embedded source bitstream using our simple, fast optimizer, we maximize expected image quality and provide for graceful degradation of the received image during fades. To achieve unequal protection, each packet is split into many Reed-Solomon symbols. The i/sup th/ symbol in each packet forms an (n,k) Reed-Solomon code or "column". A fast, nearly-optimal optimizer, based on Lagrange multipliers and optimal to within convex hull and discretization approximations, chooses k for each Reed-Solomon "column" to minimize the expected mean-square error at the receiver. We validated our use of this structure by evaluating its performance in the context of transmitting images over a wireless fading channel. The performance of this scheme was evaluated by simulating the transmission of the Lena image over a Clarke flat-fading channel with an average SNR of 10 dB and a normalized Doppler frequency of 10/sup -5/ Hz.

We address the problem of distributed source coding, i.e. compression of correlated sources that are not co-located and/or cannot communicate with each other to minimize their joint description cost. In this work we tackle the related problem of compressing a source that is correlated with another source which is available only at the decoder. In contrast to prior information-theoretic approaches, we introduce a new construction and practical framework for tackling the problem based on the judicious incorporation of channel coding principles into this source coding problem. We dub our approach as distributed source coding using syndromes (DISCUS). We focus in this paper on trellis-structured constructions of the framework to illustrate its utility. Simulation results confirm the power of DISCUS, opening up a new and exciting constructive playing-ground for the distributed source coding problem. For the distributed coding of correlated i.i.d. Gaussian sources that are noisy versions of each other with "correlation-SNR" in the range of 12 to 20 dB, the DISCUS method attains gains of 7-15 dB in SNR over the Shannon-bound using "naive" independent coding of the sources.

With the rapid growth of multimedia content in wireless communication, there is an increasing demand for efficient image and video transmission systems. We present a joint source-channel matching scheme for wireless video transmission which jointly optimizes the source and channel coder to yield the optimal transmission quality while satisfying real-time delay and buffer constraints. We utilize a parametric model approach which avoids the necessity of having detailed a priori knowledge of the coders, thus making the scheme applicable to a wide variety of source and channel coder pairs. Simulations show that the scheme yields excellent results and works for several different types of source and channel coders.

Summary form only given. With the rapid growth of wireless communications systems there is an increasing demand for efficient image and video transmission. Significant performance gains can be obtained from joint source channel matching where system resources are assigned based on the tradeoff between data and redundancy. We develop a more general approach for joint source channel matching based on a parametric distortion model that incorporates the flexibility and constraints of both the source and the channel. We use parametric models describing source and channel. We use parametric models describing source and channel characteristics that can be accurately applied to most classes of source and channel coders. To show the generality of our approach, we applied it to the familiar Said-Pearlman progressive image coder with two types of channel coders, a coder with orthogonal symbols of different power, and fixed-rate BPSK modulation overlaid with Reed Solomon codes.

Block cyclic redundancy check (CRC) codes represent a popular and powerful class of error detection techniques in modern data communication systems. Though efficient, CRCs can detect errors only after an entire block of data has been received and processed. We propose a new "continuous" error detection scheme using arithmetic coding that provides a novel tradeoff between the amount of added redundancy and the amount of time needed to detect an error once it occurs. We demonstrate how the new error detection framework improves the overall performance of transmission systems, and show how sizeable performance gains can be attained. We focus on two popular scenarios: (i) automatic repeat request (ARQ) based transmission; and (ii) forward error correction frameworks based on (serially) concatenated coding systems involving an inner error-correction code and an outer error-detection code.

We introduce a new image compression paradigm that combines compression efficiency with speed, and is based on an independent "infinite" mixture model which accurately captures the space-frequency characterization of the wavelet image representation. Specifically, we model image wavelet coefficients as being drawn from an independent generalized Gaussian distribution field, of fixed unknown shape for each subband, having zero mean and unknown slowly spatially-varying variances. Based on this model, we develop a powerful "on the fly" estimation-quantization (EQ) framework that consists of: (i) first finding the maximum-likelihood estimate of the individual spatially-varying coefficient field variances based on causal and quantized spatial neighborhood contexts; and (ii) then applying an off-line rate-distortion (R-D) optimized quantization/entropy coding strategy, implemented as a fast lookup table, that is optimally matched to the derived variance estimates. A distinctive feature of our paradigm is the dynamic switching between forward and backward adaptation modes based on the reliability of causal prediction contexts. The performance of our coder is extremely competitive with the best published results in the literature across diverse classes of images and target bitrates of interest, in both compression efficiency and processing speed. For example, our coder exceeds the objective performance of the best zerotree-based wavelet coder based on space-frequency-quantization at all bit rates for all tested images at a fraction of its complexity.

We introduce a novel region-based video compression framework that uses a morphological operation to efficiently capture motion correspondences between consecutive frames in an image sequence. Our coder is built on the observation that the motion field associated with typical image sequences can be segmented into component motion subfield "clusters" associated with distinct objects or regions in the scene, and further that these clusters can be efficiently captured using morphological operators in a "backward" framework that avoids needing to explicitly send object boundaries. Cluster matching is used in lieu of the conventional block matching methods of standard video codecs to define a cluster motion representation paradigm. Experimental coding results show about 10-20% reduction in prediction error energy and 0.3-1 db (average of about 0.44 dB) reduction in the final residue-coded peak signal to noise ratio (PSNR) using our proposed motion compensation framework on the football sequence over standard block motion compensation methods like MPEG, while also requiring less computational complexity.

Previous works, including adaptive quantizer selection and adaptive coefficient thresholding, have addressed the optimization of a baseline-decodable JPEG coder in a rate-distortion (R-D) sense. In this work, by developing an entropy-constrained quantization framework, we show that these previous works do not fully realize the attainable coding gain, and then formulate a computationally efficient way that attempts to fully realize this gain for baseline-JPEG-decodable systems. Interestingly, we find that the gains obtained using the previous algorithms are almost additive. The framework involves viewing a scalar-quantized system with fixed quantizers as a special type of vector quantizer (VQ), and then to use techniques akin to entropy-constrained vector quantization (ECVQ) to optimize the system. In the JPEG case, a computationally efficient algorithm can be derived, without training, by jointly performing coefficient thresholding, quantizer selection, and Huffman table customization, all compatible with the baseline JPEG syntax. Our algorithm achieves significant R-D improvement over standard JPEG (about 2 dB for typical images) with performance comparable to that of more complex "state-of-the-art" coders. For example, for the Lenna image coded at 1.0 bits per pixel, our JPEG-compatible coder achieves a PSNR of 39.6 dB, which even slightly exceeds the published performance of Shapiro's wavelet coder. Although PSNR does not guarantee subjective performance, our algorithm can be applied with a flexible range of visually-based distortion metrics.

Wavelet image decompositions generate a tree-structured set of coefficients, providing an hierarchical data-structure for representing images. Several recently proposed image compression algorithms have focused on new ways for exploiting dependencies between this hierarchy of wavelet coefficients. This paper presents a new framework for understanding the efficiency of one such algorithm as a simplified attempt to a global entropy-constrained image quantizer. The principle insight offered by the new framework is that improved performance is achieved by more accurately characterizing the joint probabilities of arbitrary sets of wavelet coefficients. The specific algorithm described is designed around one conveniently structured collection of such sets. The efficiency of hierarchical wavelet coding algorithms derives from their success at identifying and exploiting dependencies between coefficients in the hierarchical structure. The second part of the paper presents an empirical study of the distribution of high-band wavelet coefficients, the band responsible for most of the performance improvements of the new algorithms.>

The authors show a rate-distortion optimal quantization technique to threshold the DCT coefficients in the industry image and video coding standards JPEG and MPEG respectively. Their scheme achieves a decent thresholding gain in terms of both objective SNR (about 1 dB) as well as perceived quality and uses a fast dynamic programming recursive structure which exploits certain monotonicity characteristics of the JPEG and MPEG codebooks to drastically reduce the complexity. The primary advantage of their encoding algorithm is that it is completely compatible with the baseline JPEG and MPEG decoders.>