Large Language Models (LLMs) have transformed natural language processing, but face significant challenges in widespread deployment due to their high runtime cost. In this paper, we introduce SeedLM, a novel post-training compression method that uses seeds of a pseudo-random generator to encode and compress model weights. Specifically, for each block of weights, we find a seed that is fed into a Linear Feedback Shift Register (LFSR) during inference to efficiently generate a random matrix. This matrix is then linearly combined with compressed coefficients to reconstruct the weight block. SeedLM reduces memory access and leverages idle compute cycles during inference, effectively speeding up memory-bound tasks by trading compute for fewer memory accesses. Unlike state-of-the-art methods that rely on calibration data, our approach is data-free and generalizes well across diverse tasks. Our experiments with Llama3 70B, which is particularly challenging, show zero-shot accuracy retention at 4- and 3-bit compression to be on par with or better than state-of-the-art methods, while maintaining performance comparable to FP16 baselines. Additionally, FPGA-based tests demonstrate that 4-bit SeedLM, as model size increases, approaches a 4x speed-up over an FP16 Llama 2/3 baseline.

This paper introduces Block Data Representations (BDR), a framework for exploring and evaluating a wide spectrum of narrow-precision formats for deep learning. It enables comparison of popular quantization standards, and through BDR, new formats based on shared microexponents (MX) are identified, which outperform other state-of-the-art quantization approaches, including narrow-precision floating-point and block floating-point. MX utilizes multiple levels of quantization scaling with ultra-fine scaling factors based on shared microexponents in the hardware. The effectiveness of MX is demonstrated on real-world models including large-scale generative pretraining and inferencing, and production-scale recommendation systems.

In this paper, we propose a supervised graph representation learning method to model the relationship between brain functional connectivity (FC) and structural connectivity (SC) through a graph encoder-decoder system. The graph convolutional network (GCN) model is leveraged in the encoder to learn lower-dimensional node representations (i.e. node embeddings) integrating information from both node attributes and network topology. In doing so, the encoder manages to capture both direct and indirect interactions between brain regions in the node embeddings which later help reconstruct empirical FC networks. From node embeddings, graph representations are learnt to embed the entire graphs into a vector space. Our end-to-end model utilizes a multi-objective loss function to simultaneously learn node representations for FC network reconstruction and graph representations for subject classification. The experiment on a large population of non-drinkers and heavy drinkers shows that our model can provide a characterization of the population pattern in the SC-FC relationship, while also learning features that capture individual uniqueness for subject classification. The identified key brain subnetworks show significant between-group difference and support the promising prospect of GCN-based graph representation learning on brain networks to model human brain activity and function.

We propose a methodology to carry out vertex-frequency analyses of graph signals, with the goal of unveiling the signal's frequency occupancy over a localized region in the network. To this end, we first introduce localized graph signals in the vertex domain, by defining windows that are localized around each node by construction. Recent directed graph Fourier transform (DGFT) advances facilitate the frequency analysis of said localized signals, to reveal the signal's energy distribution in a way akin to a spectrogram in the vertex-frequency plane. We then learn a set of windows by applying gradient descent method to an optimization problem governed by penalty parameters in the spectral domain. We also argue about the tradeoff between the resolution in the vertex and frequency domains based on the said parameters. We evaluate the performance of the proposed windowed GFT approach through numerical experiments on synthetic and real-world graphs.

We study the problem of sampling a bandlimited graph signal in the presence of noise, where the objective is to select a node subset of prescribed cardinality that minimizes the signal reconstruction mean squared error (MSE). To that end, we formulate the task at hand as the minimization of MSE subject to binary constraints, and approximate the resulting NP-hard problem via semidefinite programming (SDP) relaxation. Moreover, we provide an alternative formulation based on maximizing a monotone weak submodular function and propose a randomized-greedy algorithm to find a sub-optimal subset. We then derive a worst-case performance guarantee on the MSE returned by the randomized greedy algorithm for general non-stationary graph signals. The efficacy of the proposed methods is illustrated through numerical simulations on synthetic and realworld graphs. Notably, the randomized greedy algorithm yields an order-of-magnitude speedup over state-of-the-art greedy sampling schemes, while incurring only a marginal MSE performance loss.

We address the problem of constructing a graph Fourier transform (GFT) for both undirected and directed graphs (digraphs), which decomposes graph signals into different modes of variation with respect to the underlying network. Accordingly, we seek orthonormal bases that yield maximally-spread frequency components in the graph spectral domain to better capture low, medium and high frequencies. To that end, we advocate a two-step design whereby we: (i) find the maximum directed variation (i.e., frequency on a digraph) a candidate basis vector can attain; and (ii) minimize a smooth spectral dispersion function over the achievable frequency range to obtain the desired spread GFT basis. Both steps involve non-convex, orthonormality-constrained optimization problems, which are efficiently tackled via a provably convergent, feasible optimization method on the Stiefel manifold. We illustrate the effectiveness of the novel GFT construction algorithm through numerical tests on synthetic and real-world graphs.

We address the problem of inferring an undirected graph from nodal observations, which are modeled as non-stationary graph signals generated by local diffusion dynamics on the unknown network. We propose a two-step approach where we first estimate the unknown diffusion (graph) filter, from which we recover the eigenvectors of the so-called graph-shift operator (a matrix representation of the graph). We then estimate the eigenvalues by imposing desirable properties on the graph to be recovered. To carry out the initial system identification step, we assume that second-order statistics of the inputs are available. While such quadratic filter identification problem boils down to a non-convex fourth order polynomial minimization, we propose a semidefinite relaxation with provable performance guarantees. Finally, numerical tests illustrate the use of the proposed algorithm to unveil urban mobility patterns.

We address the problem of inferring a graph from nodal observations, which are modeled as non-stationary graph signals generated by local diffusion dynamics that depend on the structure of the sought network. Using the so-called graph-shift operator (GSO) as a matrix representation of the graph, we first identify the eigenvectors of the shift matrix from realizations of the diffused signals, and then we rely on these spectral templates to estimate the eigenvalues by imposing desirable properties on the graph to be recovered. Different from the stationary setting where the GSO and the covariance matrix of the observed signals are simultaneously diagonalizable, here they are not. Hence, estimating the eigenvectors requires first estimating the unknown diffusion (graph) filter - a polynomial in the GSO which does preserve the sought eigenbasis. To carry out this initial system identification step, we leverage different sources of information on the input signal driving the diffusion process on the graph. Numerical tests showcase the effectiveness of the proposed algorithms in recovering social and structural brain graphs.