In real-world industrial scenarios, large language models (LLMs) require Continuous Learning (CL) to adapt to diverse tasks as opera- tional requirements diversify, demanding self-evolution capabilities to autonomously refine their knowledge and adapt to dynamic envi- ronments. However, existing CL approaches, such as replay-based and parameter isolation techniques, struggle with the catastrophic forgetting problem: new task training degrades performance on prior tasks due to the model's adaptation to new data distributions, which weakens its generalization to old tasks. To address this issue, we propose a novel parameter-efficient adversarial MoE framework, MoE-CL, for industrial-scale self-evolving continual instruction tuning of LLMs. Specifically, MoE-CL employs a dual-expert archi- tecture to enable self-evolution: a dedicated LoRA expert for each task to preserve task-specific knowledge, ensuring parameter inde- pendence and mitigating forgetting, and a shared LoRA expert to facilitate cross-task knowledge transfer. Specifically, a task-aware discriminator within a Generative Adversarial Network (GAN) is integrated into the shared expert to suppress task-irrelevant noise, ensuring only task-aligned knowledge is transferred during se- quential task training. Through adversarial training, the shared ex- pert learns generalized representations that mimic the task-aware discriminator, while dedicated experts retain task-specific details, balancing knowledge retention and cross-task generalization—key to the model's self-evolution by autonomously optimizing knowl- edge integration across tasks. Extensive experiments on a public MTL5 benchmark and an industrial Tencent3 benchmark validate MoE-CL's effectiveness in self-evolving continual learning. In real- world A/B testing on content compliance review in the Tencent Video Platform, MoE-CL reduced manual review costs by 15.3%,demonstrating its applicability for large-scale industrial deploy- ment where self-evolution is critical for adapting to evolving op- erational demands. Implementation code is publicly available at https://github.com/BAI-LAB/MoE-CL.

With the expansion of business scenarios, real recommender systems are facing challenges in dealing with the constantly emerging new tasks in multi-task learning frameworks. In this article, we attempt to improve the generalization ability of multi-task recommendations when dealing with new tasks. A novel two-stage prompt-tuning MTL framework (MPT-Rec) is proposed to address task irrelevance and training efficiency problems in multi-task recommender systems. Specifically, we disentangle the task-specific and task-sharing information in the multi-task pre-training stage and then use task-aware prompts to transfer knowledge from other tasks to the new task effectively. By freezing parameters in the pre-training tasks, MPT-Rec solves the negative impacts that may be brought by the new task and greatly reduces the training costs. Extensive experiments on three real-world datasets show the effectiveness of our proposed multi-task learning framework. MPT-Rec achieves the best performance compared to the SOTA multi-task learning method on three real-world datasets. Besides, it maintains comparable model performance but vastly improves the training efficiency (i.e., with up to 10% parameters in the full-training way) in the new task learning. Our code is publicly available at https://github.com/BAI-LAB/MPT-Rec .

Large Language Models (LLMs) have gained significant attention in the field of natural language processing (NLP) due to their wide range of applications. However, training LLMs for languages other than English poses significant challenges, due to the difficulty in acquiring large-scale corpus and the requisite computing resources. In this paper, we propose ChatFlow, a cross-language transfer-based LLM, to address these challenges and train large Chinese language models in a cost-effective manner. We employ a mix of Chinese, English, and parallel corpus to continuously train the LLaMA2 model, aiming to align cross-language representations and facilitate the knowledge transfer specifically to the Chinese language model. In addition, we use a dynamic data sampler to progressively transition the model from unsupervised pre-training to supervised fine-tuning. Experimental results demonstrate that our approach accelerates model convergence and achieves superior performance. We evaluate ChatFlow on popular Chinese and English benchmarks, the results indicate that it outperforms other Chinese models post-trained on LLaMA-2-7B.