Abstract To enhance the efficiency of the attention mechanism within large language models (LLMs), previous works primarily compress the Key-Value cache or group attention heads, while largely overlooking redundancy between layers. Our comprehensive analyses across various LLMs show that highly similar attention patterns persist within most layers. It’s intuitive to reduce the redundancy by sharing attention weights across layers. However, further analysis reveals two challenges: (1) Directly sharing the weight matrix without carefully rearranging the attention heads proves to be ineffective; (2) Shallow layers are vulnerable to small deviations in attention weights. Driven by these insights, we introduce LiSA, a lightweight substitute for self-attention in well-trained LLMs. LiSA employs tiny feed-forward networks to align attention heads between adjacent layers and low-rank matrices to approximate differences in layer-wise attention weights. Evaluations encompassing 13 typical benchmarks demonstrate that LiSA maintains high response quality in terms of accuracy and perplexity while reducing redundant attention calculations within 53% −84% of the total layers. Our implementations of LiSA achieve a 6 × compression of Q and K matrices within the attention mechanism, with maximum throughput improvements 19.5%, 32.3%, and 40.1% for LLaMA3-8B, LLaMA2-7B, and LLaMA2-13B, respectively. Our code is available at https://github.com/takagi97/lisa.

Yongyu Mu, Peinan Feng, Zhiquan Cao, Yuzhang Wu, Bei Li, Chenglong Wang, Tong Xiao, Kai Song, Tongran Liu, Chunliang Zhang, JingBo Zhu. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

Nowadays, with the popularity of the federated learning, it becomes crucial for us to tackle the challenges, communication cost and model robustness. And targeting at the communication bottleneck, data compression is widely used to solve the problem. Besides, the usage of variance reduction for achieving robustness and communication compression for reducing costs has been studied. The Byz-VR-MARINA pro- posed before uses random-sparsification. In this paper, we adopt the absolute compressors hard-threshold and propose a robust compressed framework Byz-VR-BARRY. Experimental results on w8a and a9a datasets have shown the effectiveness of our method, which can decrease the optimality gap obviously.

Advances in single-cell sequencing and data analysis have made it possible to infer biological trajectories spanning heterogeneous cell populations based on transcriptome variation. These trajectories yield a wealth of novel insights into dynamic processes such as development and differentiation. However, trajectory analysis relies on an assumption of trajectory continuity, and experimental limitations preclude some real-world scenarios from meeting this condition. The current lack of assessment metrics makes it difficult to ascertain if/when a given trajectory deviates from continuity, and what impact such a divergence would have on inference accuracy is unclear. By analyzing simulated breaks introduced into in silico and real single-cell data, we found that discontinuity caused precipitous drops in the accuracy of trajectory inference. We then generate a simple scoring algorithm for assessing trajectory continuity, and found that continuity assessments in real-world cases of intestinal stem cell development and CD8 + T cells differentiation efficiently identifies trajectories consistent with empirical knowledge. This assessment approach can also be used in cases where a priori knowledge is lacking to screen a pool of inferred lineages for their adherence to presumed continuity, and serve as a means for weighing higher likelihood trajectories for validation via empirical studies, as exemplified by our case studies in psoriatic arthritis and acute kidney injury. This tool is freely available through github at qingshanni/scEGRET.

Recent advances in bioinformatics analyses have led to the development of novel tools enabling the capture and trajectory mapping of single-cell RNA sequencing (scRNAseq) data. However, there is a lack of methods to assess the contributions of biological pathways and transcription factors to an overall developmental trajectory mapped from scRNAseq data. In this manuscript, we present a simplified approach for trajectory inference of pathway significance (TIPS) that leverages existing knowledgebases of functional pathways and other gene lists to provide further mechanistic insights into a biological process. TIPS identifies key pathways which contribute to a process of interest, as well as the individual genes that best reflect these changes. TIPS also provides insight into the relative timing of pathway changes, as well as a suite of visualizations to enable simplified data interpretation of scRNAseq libraries generated using a wide range of techniques. The TIPS package can be run through either a web server or downloaded as a user-friendly GUI run in R, and may serve as a useful tool to help biologists perform deeper functional analyses and visualization of their single-cell data.