Large Language Models (LLMs) have advanced rapidly but face significant memory demands.While quantization could alleviate the memory-bound issue, current methods typically require lengthy training to recover accuracy under low bit width.In that circumstance, deployment across scenarios with different resource constraints necessitates repeated training, amplifying the issue of protracted training.It is beneficial to train a once-for-all (OFA) supernet capable of offering optimal subnets for downstream applications.To extend the oncefor-all setting to LLMs, we decouple the shared weights to mitigate the interference and integrate Low-Rank adapters to enhance training efficiency.Furthermore, it is observed that there is an imbalance in the allocation of training resources due to traditional uniform sampling.A non-parametric scheduler is introduced to adjust the sampling rate for each quantization configuration, thereby achieving a more balanced allocation among subnets with varying demands.We validate the approach on LLaMA families and Mistral on downstream evaluation, demonstrating high performance while significantly reducing deployment time faced with multiple scenarios.1

Large language models have demonstrated promising capabilities upon scaling up parameters. However, serving large language models incurs substantial computation and memory movement costs due to their large scale. Quantization methods have been employed to reduce service costs and latency. Nevertheless, outliers in activations hinder the development of INT4 weight-activation quantization. Existing approaches separate outliers and normal values into two matrices or migrate outliers from activations to weights, suffering from high latency or accuracy degradation. Based on observing activations from large language models, outliers can be classified into channel-wise and spike outliers. In this work, we propose Rotated Runtime Smooth (**RRS**), a plug-and-play activation smoother for quantization, consisting of Runtime Smooth and the Rotation operation. Runtime Smooth (**RS**) is introduced to eliminate **channel-wise outliers** by smoothing activations with channel-wise maximums during runtime. The Rotation operation can narrow the gap between **spike outliers** and normal values, alleviating the effect of victims caused by channel-wise smoothing. The proposed method outperforms the state-of-the-art method in the LLaMA and Qwen families and improves WikiText-2 perplexity from 57.33 to 6.66 for INT4 inference.

Convolutional Neural Networks (CNNs) have ex-hibited great power in various vision tasks. However, the lack of transform-invariant property limits their further applications in complicated real-world scenarios. In this work, we pro-posed a novel generalized one-dimension convolutional operator (OneDConv), which dynamically transforms the convolution kernels based on the input features in a computationally and parametrically efficient manner. The proposed operator can extract the transform-invariant features naturally. It improves the robustness and generalization of convolution without sac-rificing the performance of common images. The proposed OneDConv operator can substitute the vanilla convolution. Thus, it can readily be incorporated into popular convolutional architectures, supporting end-to-end training. Empirical evaluations on popular benchmarks reveal OneDConv's superior performance over the standard convolution and competitive models in handling canonical and distorted images.

Large-scale pre-training has shown great potential to enhance models on downstream tasks in vision and language. Developing similar techniques for scalp electroencephalogram (EEG) is suitable since unlabelled data is plentiful. Meanwhile, various sampling channel selections and inherent structural and spatial information bring challenges and avenues to improve existing pre-training strategies further. In order to break boundaries between different EEG resources and facilitate cross-dataset EEG pre-training, we propose to map all kinds of channel selections to a unified topology. We further introduce MMM, a pre-training framework with Multi-dimensional position encoding, Multi-level channel hierarchy, and Multi-stage pre-training strategy built on the unified topology to obtain topology-agnostic representations. Experiments demonstrate that our approach yields impressive improvements over previous state-of-the-art techniques on emotional recognition benchmark datasets.