The remarkable success of Large Language Models (LLMs) relies heavily on their substantial scale, which poses significant challenges during model deployment in terms of latency and memory consumption. Recently, numerous studies have attempted to compress LLMs using one-shot pruning methods. However, these methods often suffer from considerable performance degradation on complex language understanding tasks, raising concerns about the feasibility of pruning in LLMs. To address this issue, we propose Adaptive Sparse Trainer (AST), a novel and efficient retraining framework tailored for semi-structured sparse models. AST enables models to learn optimal masks during the weight update process without incurring additional computational overhead. Furthermore, we demonstrate that incorporating knowledge distillation significantly improves retraining efficiency and enhances model performance under fixed computational constraints. Additionally, a supplementary set of well-initialized parameters is integrated to further augment the model's efficacy. AST achieves state-of-the-art performance with minimal training cost. When applied to the LLaMA2-7B model, AST reduces the perplexity and zero-shot accuracy gap between dense and 2:4 semi-structured sparse models to 0.6 and 1.16%, respectively, utilizing less than 0.4% of the pretraining tokens and GPU hours. Our work demonstrates the feasibility of deploying semi-structured sparse LLMs and offers a promising alternative for achieving highly compressed models when combined with existing quantization techniques.

Training large transformers is slow, but recent innovations on GPU architecture give us an advantage. NVIDIA Ampere GPUs can execute a fine-grained 2:4 sparse matrix multiplication twice as fast as its dense equivalent. In the light of this property, we comprehensively investigate the feasibility of accelerating feed-forward networks (FFNs) of transformers in pre-training. First, we define a “flip rate” to monitor the stability of a 2:4 training process. Utilizing this metric, we propose three techniques to preserve accuracy: to modify the sparse-refined straight-through estimator by applying the masked decay term on gradients, to determine a feasible decay factor in warm-up stage, and to enhance the model’s quality by a dense fine-tuning procedure near the end of pre-training. Besides, we devise two techniques to practically accelerate training: to calculate transposable 2:4 masks by convolution, and to accelerate gated activation functions by reducing GPU L2 cache miss. Experiments show that our 2:4 sparse training algorithm achieves similar convergence to dense training algorithms on several transformer pre-training tasks, while actual acceleration can be observed on different shapes of transformer block apparently. Our toolkit is available at https://github.com/huyz2023/2by4-pretrain.

Modern neuroimaging techniques offer disctinct views on brain structure and function. Data acquired using these techniques can be analyzed in terms of its network structure to identify organizing principles at the systems level. Graph representations are flexible frameworks where nodes are related to brain regions and edges to structural or functional links. Most research to date has focused on analyzing these graphs reflecting structure or function. Graph signal processing (GSP) is an emerging area of research where signals at the nodes are studied atop the underlying graph structure. Here, we review GSP tools for brain imaging data and discuss their potential to integrate brain structure with function. We discuss how brain activity can be meaningfully filtered. We also derive surrogate data as a null model to test significance for graph signals. We review that individuals with less concentration on graph high frequency could switch attention faster.

This paper develops new designs for recommender systems inspired by recent advances in graph signal processing. Recommender systems aim to predict unknown ratings by exploiting the information revealed in a subset of user-item observed ratings. Leveraging the notions of graph frequency and graph filters, we demonstrate that linear latent factor models, such as low-rank matrix completion, can be viewed as bandlimited interpolation algorithms that operate in a frequency domain given by the spectrum of a joint user and item network. This new interpretation paves the way to new methods for enhanced rating prediction. We propose a low complexity method by exploiting the eigenvector of correlation matrices constructed from known ratings. In the MovieLens 100k dataset, our designs reduce the root mean squared error compared to the ones in benchmark matrix completion by 0.6% and benchmark nearest neighbor methods by 4.2%.

Modern neuroimaging techniques provide us with unique views on brain structure and function; i.e., how the brain is wired, and where and when activity takes place. Data acquired using these techniques can be analyzed in terms of its network structure to reveal organizing principles at the systems level. Graph representations are versatile models where nodes are associated to brain regions and edges to structural or functional connections. Structural graphs model neural pathways in white matter, which are the anatomical backbone between regions. Functional graphs are built based on functional connectivity, which is a pairwise measure of statistical interdependency between pairs of regional activity traces. Therefore, most research to date has focused on analyzing these graphs reflecting structure or function. Graph signal processing (GSP) is an emerging area of research where signals recorded at the nodes of the graph are studied atop the underlying graph structure. An increasing number of fundamental operations have been generalized to the graph setting, allowing to analyze the signals from a new viewpoint. Here, we review GSP for brain imaging data and discuss their potential to integrate brain structure, contained in the graph itself, with brain function, residing in the graph signals. We review how brain activity can be meaningfully filtered based on concepts of spectral modes derived from brain structure. We also derive other operations such as surrogate data generation or decompositions informed by cognitive systems. In sum, GSP offers a novel framework for the analysis of brain imaging data.

This paper presents methods to analyze functional brain networks and signals from graph spectral perspectives. The notion of frequency and filters recently generalized to irregular graph domains defines brain graph frequencies associated with different levels of spatial smoothness across the brain regions. Brain network frequency also enables the decomposition of brain signals into pieces corresponding to smooth or rapid variations. The methods are utilized to analyze brain networks and signals as subjects master a simple motor skill. We observe that brain signals corresponding to different graph frequencies exhibit different levels of contribution to active learning. Specifically, we notice a strong association between graph spectral properties of brain networks and the level of exposure to tasks performed, and recognize the most contributing and important frequency signatures at different task familiarity.

This paper examines metric spaces in which the distance between any pair of nodes is given by an interval. The goal is to investigate methods for hierarchical clustering, i.e., a family of nested partitions indexed by a connectivity parameter, deduced from the underlying distance intervals of the metric spaces. Our construction is based on designing admissible methods abiding to the axioms of value and transformation. Two admissible methods are constructed and are shown to provide upper and lower bounds in the space of all admissible methods. Practical implications are explored by clustering moving points via snapshots. The proposed clustering methods succeed in identifying underlying clustering structures via the maximum and minimum distances in all snapshots.

High order networks are weighted complete hypergraphs collecting relationships between elements of tuples. Valid metric distances between high order networks have been defined but they are difficult to compute when the number of nodes is large. We relate high order networks to the filtrations of simplicial complexes and show that the distance between networks can be lower bounded by the difference between the homological features of their respective filtrations. Practical implications are explored by comparing the coauthorship networks of engineering and mathematics academic journals. The lower bounds succeed in discriminating engineering communities from mathematics and in differentiating engineering communities with different research interests.

This paper presents diffusion filtering as a method to smooth signals defined on the nodes of a graph or network. Diffusion filtering considers the given signals as initial temperature distributions in the nodes and diffuses heat through the edges of the graph. The filtered signal is determined by the accumulated temperatures over time at each node. We show multiple other interpretations of diffusion filtering and describe how it can be generalized to encompass a wide class of networks making it suitable for real-world applications. We prove that diffused signals are stable to perturbations in the underlying network. Further, we demonstrate how diffusion filtering can be applied to improve the performance of recommendation systems by considering the problem of predicting ratings from a signal processing perspective.

This paper presents two families of distances in the space of high order proximity networks. The distances measure differences between networks and are shown to be valid metrics in the space of high order proximity networks modulo permutation isomorphisms. Practical implications are explored by comparing the coauthorship networks of two popular signal processing researchers. The metrics succeed in identifying their respective collaboration patterns.