We present a novel rank-sliced GAther-Scatter Activation (GASA) algorithm to minimize I/O costs in computing neural network layer activations (XW) between a data matrix X, and a singular value decomposition (SVD) of a weight matrix W = UΣVT. We maintain high accuracy with ResNet-18 [1] on the CIFAR-10 [2] dataset and Deberta-V3-base [3] on the IMDB dataset [4] at high sparsities (i.e., up to 85% sparsity for U and V ) by using rank pruning for W and UV pruning for each rank slice. Furthermore, with rank-sliced computation, we can perform parameter-efficient fine-tuning on the resulting sparse networks while preserving sparsity to retain the sparsity-induced computational efficiency for inference. That is, our rank-sliced weight update preserves the original sparsity structure of each W. Our sparsity-preserving fine-tuning maintains model accuracy under adapter ranks as low as 8, compared to the rank of 150 of the pre-trained pruned model.

We offer a novel approach to matrix-matrix multiplication computation on computing platforms with memory hierarchies. Constant-bandwidth (CB) blocks improve computation throughput for architectures limited by external memory bandwidth. Configuring the shape and size of CB blocks operating from within any memory hierarchy level (e.g., internal SRAM), we achieve high throughput while holding external bandwidth (e.g., with DRAM) constant. We explain how, surprisingly, CB blocks can maintain constant external bandwidth as computation throughput increases. Analogous to partitioning a cake into pieces, we dub our CB-partitioned system CAKE.