The mechanisms behind multilingual capabilities in Large Language Models (LLMs) have been examined using neuron-based or internal-activation-based methods.However, these methods often face challenges such as superposition and layer-wise activation variance, which limit their reliability.Sparse Autoencoders (SAEs) offer a more nuanced analysis by decomposing the activations of LLMs into a sparse linear combination of SAE features.We introduce a novel metric to assess the monolinguality of features obtained from SAEs, discovering that some features are strongly related to specific languages.Additionally, we show that ablating these SAE features only significantly reduces abilities in one language of LLMs, leaving others almost unaffected.Interestingly, we find some languages have multiple synergistic SAE features, and ablating them together yields greater improvement than ablating individually.Moreover, we leverage these SAE-derived language-specific features to enhance steering vectors, achieving control over the language generated by LLMs.The code is publicly available at https://github.com/ Aatrox103/multilingual-llm-features.

Yidan Zhang, Yu Wan, Boyi Deng, Baosong Yang, Hao-Ran Wei, Fei Huang, Bowen Yu, Dayiheng Liu, Junyang Lin, Fei Huang, Jingren Zhou. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

Recently, large language models (LLMs) have showcased remarkable capabilities across a diverse range of applications, including general natural language processing (NLP) and domain-specific tasks. Empirical evidence indicates that LLMs have matched or even surpassed human performance in various areas, such as language translation, reading comprehension, and logical reasoning. However, preliminary research reveals that LLMs struggle with basic character and word editing, which is crucial for practical tasks such as creating 1000-word articles or modifying specific text information. To comprehensively assess the capabilities of LLMs in character and word understanding and manipulation (CWUM), we introduce the CWUM benchmark in Chinese and English. CWUM comprises 23 tasks focusing on text editions, including counting, identification, insertion, and reversal. A comprehensive evaluation of nine advanced LLMs on CWUM is conducted, which highlights significant failures of existing LLMs on CWUM tasks that humans can solve perfectly with 100% accuracy. Meanwhile, specific deficiencies of LLMs in basic language understanding and manipulation are revealed by performing quality and quantity analysis. Furthermore, in the experiment part, various methods are investigated to improve model performance, demonstrating the effectiveness of supervised fine-tuning (SFT) in enhancing model performance on CWUM while maintaining generalization abilities on unseen tasks.

Aiming to identify entity pairs with biomedical semantic relations and assign specific relation types, biomedical relation extraction (BioRE) plays a critical role in biomedical text mining and information extraction (IE). Recent studies indicate that general large language models (LLMs) have made some breakthroughs in general relation extraction (RE) tasks. However, even the advanced open-source LLMs struggle with BioRE tasks. For example, WizardLM-70B and LLaMA-2-70B achieve F-scores of 14.05 and 12.21 on the BioRED dataset, respectively, significantly lagging behind the state-of-the-art (SOTA) method which scores 65.17. To address this gap, a multitask instruction-tuning framework is proposed, which can transform general LLMs into BioRE-specialized models with our meticulously curated instruction dataset, REInstruct, comprising 150000 diverse and quality instruction-response pairs. Consequently, we introduce REaMA, a series of open-source LLMs with sizes of 7B and 13B specifically tailored for BioRE tasks. Experimental results on seven representative BioRE datasets show that both REaMA-2-7B and REaMA-2-13B acquire promising performance on all datasets. Remarkably, the larger REaMA-2-13B outperforms the current SOTA method on five out of seven datasets. The result exhibits the effectiveness of instruction-tuning on REInstruct in eliciting strong RE capabilities in LLMs. Furthermore, we show that incorporating chain of thought (CoT) into REInstruct can further enhance the generalization ability of REaMA. The project is available at https://github.com/stzpp/REaMA.

Mingfeng Xue, Dayiheng Liu, Wenqiang Lei, Jie Fu, Jian Lan, Mei Li, Baosong Yang, Jun Xie, Yidan Zhang, Dezhong Peng, Jiancheng Lv. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

Abstract Recently, many studies have been carried out on model compression to handle the high computational cost and high memory footprint brought by the implementation of deep neural networks. In this paper, model compression of convolutional neural networks is constructed as a multiobjective optimization problem with two conflicting objectives, reducing the model size and improving the performance. A novel structured pruning method called Conventional-based and Evolutionary Approaches Guided Multiobjective Pruning (CEA-MOP) is proposed to address this problem, where the power of conventional pruning methods is effectively exploited for the evolutionary process. A delicate balance in pruning rate and model accuracy has been automated achieved by a multiobjective optimization evolutionary model. First, an ensemble framework integrates pruning metrics to establish a codebook for further evolutionary operations. Then, an efficient coding method is developed to shorten the length of chromosome, thus ensuring its superior scalability. Finally, sensitivity analysis is automatically carried out to determine the upper bound of pruning rate for each layer. Notably, on CIFAR-10, CEA-MOP reduces more than 50% FLOPs on ResNet-110 and improves the relative accuracy. Moreover, on ImageNet, CEA-MOP reduces more than 50% FLOPs on ResNet-101 with negligible top-1 accuracy drop.

Filter pruning has been widely applied on compression of deep convolutional neural networks. Especially, the filter pruning approach using evolutionary algorithms has exhibited powerful capabilities of automatic model compression and global optimal solution search. However, the exponential increasing of search space brought by the extremely large scale of deep networks makes it difficult to achieve a well-compressed model with limited time and energy. In this paper, we design an effective method aiming at improving the efficiency in evolutionary pruning, called IEEPruner. Multiple pruning criteria are first utilized to initialize an efficient search space, and then a novel encoding scheme is proposed to further reduce the search space. After that, we formulate the search of the optimal pruned structure as a multiobjective optimization problem, and integrate a multiobjective evolutionary algorithm to achieve a good balance between model size and accuracy in an automatic manner. Thus, we automatically search for the optimal pruned model in this search space. Extensive experiments on widely used convolutional neural networks including VGGNet, ResNet, and LeNet have demonstrated the efficiency of the proposed evolutionary pruning method.

Typical methods for unsupervised text style transfer often rely on two key ingredients: 1) seeking the explicit disentanglement of the content and the attributes, and 2) troublesome adversarial learning. In this paper, we show that neither of these components is indispensable. We propose a new framework that utilizes the gradients to revise the sentence in a continuous space during inference to achieve text style transfer. Our method consists of three key components: a variational auto-encoder (VAE), some attribute predictors (one for each attribute), and a content predictor. The VAE and the two types of predictors enable us to perform gradient-based optimization in the continuous space, which is mapped from sentences in a discrete space, to find the representation of a target sentence with the desired attributes and preserved content. Moreover, the proposed method naturally has the ability to simultaneously manipulate multiple fine-grained attributes, such as sentence length and the presence of specific words, when performing text style transfer tasks. Compared with previous adversarial learning based methods, the proposed method is more interpretable, controllable and easier to train. Extensive experimental studies on three popular text style transfer tasks show that the proposed method significantly outperforms five state-of-the-art methods.