Deep neural networks (DNNs), despite their performance on a wide variety of tasks, are still out of reach for many applications as they require significant computational resources. In this paper, we present a low-rank based end-to-end deep neural network compression framework with the goal of enabling DNNs performance to computationally constrained devices. The proposed framework includes techniques for low-rank based structural approximation, quantization and lossless arithmetic coding. Many of these techniques have been accepted in the MPEG working draft on compressed Neural Network Representations. We demonstrate the efficacy of the proposed framework via extensive experiments on a variety of DNNs for various tasks considered in this standardization activity. These techniques provide impressive performance on DNNs used in ImageNet Large-Scale Visual Recognition Challenge by compressing VGG16 by 61x, ResNet50 by almost 15x, and MobileNetV2 by almost 7x.

Many natural signals lie in a union of subspaces, which we can exploit when compressing these signals to maintain a high level of fidelity while significantly reducing the storage size. Standard compression techniques for natural signals such as images follow a general pipeline which uses predetermined transformations to encode and decode data. Recent advances in deep learning-based techniques for compressing image data have shown results which compete with existing compression standards. Inspired by the success of these deep learning methods, we evaluate various classical and deep learning-based methods for encoding and decoding signals which follow a union-of-subspaces structure. On the classical side, we evaluate compressed sensing with a learned dictionary, whereas for deep learning-based techniques, we consider an autoencoder and a deep generative model-based variant of compressed sensing. Our results suggest that while classical compressed sensing-based methods work well, deep learning-based techniques perform better as the union-of-subspaces signal structure becomes more complex.