Monolithic large language models (LLMs) like GPT-4 have paved the way for modern generative AI applications. Training, serving, and maintaining monolithic LLMs at scale, however, remains prohibitively expensive and challenging. The disproportionate increase in compute-to-memory ratio of modern AI accelerators have created a memory wall, necessitating new methods to deploy AI. Recent research has shown that a composition of many smaller expert models, each with several orders of magnitude fewer parameters, can match or exceed the capabilities of monolithic LLMs. Composition of Experts (CoE) is a modular approach that lowers the cost and complexity of training and serving. However, this approach presents two key challenges when using conventional hardware: (1) without fused operations, smaller models have lower operational intensity, which makes high utilization more challenging to achieve; and (2) hosting a large number of models can be either prohibitively expensive or slow when dynamically switching between them. In this paper, we describe how combining CoE, streaming dataflow, and a three-tier memory system scales the AI memory wall. We describe Samba-CoE, a CoE system with 150 experts and a trillion total parameters. We deploy Samba-CoE on the SambaNova SN40L Reconfigurable Dataflow Unit (RDU) -a commercial dataflow accelerator architecture that has been codesigned for enterprise inference and training applications. The chip introduces a new three-tier memory system with on-chip distributed SRAM, on-package HBM, and off-package DDR DRAM. A dedicated inter-RDU network enables scaling up and out over multiple sockets. We demonstrate speedups ranging from 2× to 13× on various benchmarks running on eight RDU sockets compared with an unfused baseline. We show that for CoE inference deployments, the 8-socket RDU Node reduces machine footprint by up to 19 ×, speeds up model switching time by 15× to 31×, and achieves an overall speedup of 3.7× over a DGX H100 and 6.6× over a DGX A100.

Neural Network Coding and Representation (NNR) is the first international standard for efficient compression of neural networks (NNs). The standard is designed as a toolbox of compression methods, which can be used to create coding pipelines. It can be either used as an independent coding framework (with its own bitstream format) or together with external neural network formats and frameworks. For providing the highest degree of flexibility, the network compression methods operate per parameter tensor in order to always ensure proper decoding, even if no structure information is provided. The NNR standard contains compression-efficient quantization and deep context-adaptive binary arithmetic coding (DeepCABAC) as core encoding and decoding technologies, as well as neural network parameter pre-processing methods like sparsification, pruning, low-rank decomposition, unification, local scaling and batch norm folding. NNR achieves a compression efficiency of more than 97% for transparent coding cases, i.e. without degrading classification quality, such as top-1 or top-5 accuracies. This paper provides an overview of the technical features and characteristics of NNR.

Deep neural networks (DNNs), despite their performance on a wide variety of tasks, are still out of reach for many applications as they require significant computational resources. In this paper, we present a low-rank based end-to-end deep neural network compression framework with the goal of enabling DNNs performance to computationally constrained devices. The proposed framework includes techniques for low-rank based structural approximation, quantization and lossless arithmetic coding. Many of these techniques have been accepted in the MPEG working draft on compressed Neural Network Representations. We demonstrate the efficacy of the proposed framework via extensive experiments on a variety of DNNs for various tasks considered in this standardization activity. These techniques provide impressive performance on DNNs used in ImageNet Large-Scale Visual Recognition Challenge by compressing VGG16 by 61x, ResNet50 by almost 15x, and MobileNetV2 by almost 7x.

Many natural signals lie in a union of subspaces, which we can exploit when compressing these signals to maintain a high level of fidelity while significantly reducing the storage size. Standard compression techniques for natural signals such as images follow a general pipeline which uses predetermined transformations to encode and decode data. Recent advances in deep learning-based techniques for compressing image data have shown results which compete with existing compression standards. Inspired by the success of these deep learning methods, we evaluate various classical and deep learning-based methods for encoding and decoding signals which follow a union-of-subspaces structure. On the classical side, we evaluate compressed sensing with a learned dictionary, whereas for deep learning-based techniques, we consider an autoencoder and a deep generative model-based variant of compressed sensing. Our results suggest that while classical compressed sensing-based methods work well, deep learning-based techniques perform better as the union-of-subspaces signal structure becomes more complex.

In this paper, we propose MCformer - a novel deep neural network for the automatic modulation classification task of complex-valued raw radio signals. MCformer architecture leverages convolution layer along with self-attention based encoder layers to efficiently exploit temporal correlation between the embeddings produced by convolution layer. MCformer provides state of the art classification accuracy at all signal-to-noise ratios in the RadioML2016.10b data-set with significantly less number of parameters which is critical for fast and energy-efficient operation.

Techniques which leverage channel state information (CSI) at a transmitter to adapt wireless signals to changing propagation conditions have been shown to improve the reliability of modern multiple input multiple output (MIMO) communication systems. To reduce overhead, previous works have proposed to compress CSI matrices using a trained deep autoencoder (AE) at the receiver before feeding it back to the transmitter, and recent work has proposed to quantize and perform entropy coding on the compressed CSI to further reduce communication complexity. While these methods are effective, they either do not incorporate quantization and lossless coding into their end-to-end optimization, or do not achieve performance comparable to methods that do not use quantization and entropy coding. In this work, we propose a new AE-based feedback method which uses an entropy bottleneck layer to both quantize and losslessly code the compressed CSI. This bottleneck layer allows us to jointly optimize bit-rate and distortion to achieve a highly-compressed CSI representation which preserves important channel information. Our method achieves better reconstruction quality than existing autoencoder-based CSI feedback methods for a wide range of bit-rates on simulated data, in both indoor and outdoor wireless settings.

Despite many modern applications of Deep Neural Networks (DNNs), the large number of parameters in the hidden layers makes them unattractive for deployment on devices with storage capacity constraints. In this paper we propose a Data-Driven Low-rank (DDLR) method to reduce the number of parameters of pretrained DNNs and expedite inference by imposing low-rank structure on the fully connected layers, while controlling for the overall accuracy and without requiring any retraining. We pose the problem as finding the lowest rank approximation of each fully connected layer with given performance guarantees and relax it to a tractable convex optimization problem. We show that it is possible to significantly reduce the number of parameters in common DNN architectures with only a small reduction in classification accuracy. We compare DDLR with Net-Trim, which is another data-driven DNN compression technique based on sparsity and show that DDLR consistently produces more compressed neural networks while maintaining higher accuracy.

Despite their well-documented learning capabilities in clean environments, deep convolutional neural networks (CNNs) are extremely fragile in adversarial settings, where carefully crafted perturbations created by an attacker can easily disrupt the task at hand. Numerous methods have been proposed for designing effective attacks, while the design of effective defense schemes is still an open area. This work leverages randomization-based defense schemes to introduce a sampling mechanism for strong and efficient defense. To this end, sampling is proposed to take place over the matricized mid-layer data in the neural network, and the sampling probabilities are systematically obtained via variance minimization. The proposed defense only requires adding sampling blocks to the network in the inference phase without extra overhead in the training. In addition, it can be utilized on any pre-trained network without altering the weights. Numerical tests corroborate the improved defense against various attack schemes in comparison with state-of-the-art randomized defenses.

Several complex tasks that arise in organizations can be simplified by mapping them into a matrix completion problem. In this paper, we address a key challenge faced by our company: predicting the efficiency of artists in rendering visual effects (VFX) in film shots. We tackle this challenge by using a two-fold approach: first, we transform this task into a constrained matrix completion problem with entries bounded in the unit interval [0,1]; second, we propose two novel matrix factorization models that leverage our knowledge of the VFX environment. Our first approach, expertise matrix factorization (EMF), is an interpretable method that structures the latent factors as weighted user-item interplay. The second one, survival matrix factorization (SMF), is instead a probabilistic model for the underlying process defining employees' efficiencies. We show the effectiveness of our proposed models by extensive numerical tests on our VFX dataset and two additional datasets with values that are also bounded in the [0,1] interval.

This paper considers the problem of estimating an unknown high dimensional signal from (typically low-dimensional) noisy linear measurements, where the desired unknown signal is assumed to possess a group-sparse structure, i.e. given a (pre-defined) partition of its entries into groups, only a small number of such groups are non-zero. Assuming the unknown group-sparse signal is generated according to a certain statistical model, we provide guarantees under which it can be efficiently estimated via solving the well-known group Lasso problem. In particular, we demonstrate that the set of indices for non-zero groups of the signal (called the group-level support of the signal) can be exactly recovered by solving the proposed group Lasso problem provided that its constituent non-zero groups are small in number and possess enough energy. Our guarantees rely on the well-conditioning of measurement matrix, which is expressed in terms of the block coherence parameter and can be efficiently computed. Our results are non-asymptotic in nature and therefore applicable to practical scenarios.

This paper examines the existence of efficiently implementable approximations of a general real linear dimensionality reduction (LDR) operator. The specific focus is on approximating a given LDR operator with a partial circulant structured matrix (a matrix whose rows are related by circular shifts) as these constructions allow for low-memory footprint and computationally efficient implementations. Our main contributions are theoretical: we quantify how well general matrices may be approximated (in a Frobenius sense) by partial circulant structured matrices, and also consider a variation of this problem where the aim is only to accurately approximate the action of a given LDR operator on a restricted set of inputs. For the latter setting, we also propose a sparsity-regularized alternating minimization based algorithm for learning partial circulant approximations from data, and provide experimental evidence demonstrating the potential efficacy of this approach on real-world data.

“To be considered for the 2017 IEEE Jack Keil Wolf ISIT Student Paper Award.” In this paper we study the problem of noisy tensor completion for tensors that admit a canonical polyadic or CANDE-COMP/PARAFAC (CP) decomposition with one of the factors being sparse. We present general theoretical error bounds for an estimate obtained by using a complexity-regularized maximum likelihood principle and then instantiate these bounds for the case of additive white Gaussian noise. We also provide an ADMM-type algorithm for solving the complexity-regularized maximum likelihood problem and validate the theoretical finding via experiments on synthetic data set.

Inductive Matrix Completion (IMC) is an important class of matrix completion problems that allows direct inclusion of available features to enhance estimation capabilities. These models have found applications in personalized recommendation systems, multilabel learning, dictionary learning, etc. This paper examines a general class of noisy matrix completion tasks where the underlying matrix is following an IMC model i.e., it is formed by a mixing matrix (a priori unknown) sandwiched between two known feature matrices. The mixing matrix here is assumed to be well approximated by the product of two sparse matrices - referred here to as “sparse factor models.” We leverage the main theorem of [1] and extend it to provide theoretical error bounds for the sparsity-regularized maximum likelihood estimators for the class of problems discussed in this paper. The main result is general in the sense that it can be used to derive error bounds for various noise models. In this paper, we instantiate our main result for the case of Gaussian noise and provide corresponding error bounds in terms of squared loss.

This paper examines a general class of noisy matrix completion tasks, where the goal is to estimate a matrix from observations obtained at a subset of its entries, each of which is subject to random noise or corruption. Our specific focus is on settings where the matrix to be estimated is well-approximated by a product of two (a priori unknown) matrices, one of which is sparse. Such structural models-referred to here as sparse factor models-have been widely used, for example, in subspace clustering applications, as well as in contemporary sparse modeling and dictionary learning tasks. Our main theoretical contributions are estimation error bounds for sparsity-regularized maximum likelihood estimators for the problems of this form, which are applicable to a number of different observation noise or corruption models. Several specific implications are examined, including scenarios where observations are corrupted by additive Gaussian noise or additive heavier-tailed (Laplace) noise, Poisson-distributed observations, and highly quantized (e.g., 1 b) observations. We also propose a simple algorithmic approach based on the alternating direction method of multipliers for these tasks, and provide experimental evidence to support our error analyses.

Multicell cooperative processing with limited backhaul traffic is studied for cellular uplinks. Aiming at reduced backhaul overhead, a sparse multicell linear receive-filter design problem is formulated. Both unstructured distributed cooperation and clustered cooperation, in which base station groups are formed for tight cooperation, are considered. Dynamic clustered cooperation, where the sparse equalizer and the cooperation clusters are jointly determined, is solved via alternating minimization based on spectral clustering and group-sparse regression. Furthermore, decentralized implementations of both unstructured and clustered cooperation schemes are developed for scalability, robustness, and computational efficiency. Extensive numerical tests verify the efficacy of the proposed methods.