Large language models (LLMs) are increasingly powering web-based applications, whose effectiveness relies on fine-tuning with large-scale instruction data. However, such data often contains valuable or sensitive information that limits its public sharing among business organizations. Federated learning (FL) enables collaborative fine-tuning of LLMs without accessing raw data. Existing approaches to federated LLM fine-tuning usually adopt a uniform model architecture, making it challenging to fit highly heterogeneous client-side data in varying domains and tasks, e.g., hospitals and financial institutions conducting federated fine-tuning may require different LLM architectures due to the distinct nature of their domains and tasks. To address this, we propose FedAMoLE, a lightweight personalized FL framework that enables data-driven heterogeneous model architectures. It features a heterogeneous mixture of low-rank adaptation (LoRA) experts module to aggregate architecturally heterogeneous models and a reverse selection-based expert assignment strategy to tailor model architectures for each client based on data distributions. Experiments across seven scenarios demonstrate that FedAMoLE improves client-side performance by an average of 5.97% over existing approaches while maintaining practical memory, communication, and computation overhead.

With the development of Federated Learning (FL) in IoT Edge-Cloud Computing environments, mobile terminals are able to cooperate without the leakage on raw data. However, factors including the terminals' high mobility and the network fluctuations make the cooperator selection during FL training extremely complex. Under the distributed cooperation, traditional FL strategies show certain limitations and cannot always select the available nodes when training, leading to the difficulties in energy and latency optimization. In this paper, we propose a Hierarchical Reinforcement Learning (HRL)-based federated knowledge distillation (HRL-FedKD) framework in which both high-level and low-level controllers utilize the Double Deep Q-Network (DDQN) algorithm. The high-level controller selects the nodes participating in FL training, while the low-level controller determines the number of local training epochs for each node. After training, the global model will be compressed into a lightweight model by knowledge distillation (KD) in deployment while preserving the personalization of local models. The experiments were conducted using Chest X-Ray and Brain Tumor MRI datasets to validate the proposed FL strategy. The results demonstrate that the HRL-FedKD framework can effectively optimize latency and energy consumption in complex state spaces.

Yunkun Wang, Yue Zhang, Zhen Qin, Chen Zhi, Binhua Li, Fei Huang, Yongbin Li, Shuiguang Deng. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Semi-Supervised Learning (SSL) can leverage abundant unlabeled data to boost model performance. However, the class-imbalanced data distribution in real-world scenarios poses great challenges to SSL, resulting in performance degradation. Existing class-imbalanced semi-supervised learning (CISSL) methods mainly focus on rebalancing datasets but ignore the potential of using hard examples to enhance performance, making it difficult to fully harness the power of unlabeled data even with sophisticated algorithms. To address this issue, we propose a method that enhances the performance of Imbalanced Semi-Supervised Learning by Mining Hard Examples (SeMi). This method distinguishes the entropy differences among logits of hard and easy examples, thereby identifying hard examples and increasing the utility of unlabeled data, better addressing the imbalance problem in CISSL. In addition, we maintain a class-balanced memory bank with confidence decay for storing high-confidence embeddings to enhance the pseudo-labels' reliability. Although our method is simple, it is effective and seamlessly integrates with existing approaches. We perform comprehensive experiments on standard CISSL benchmarks and experimentally demonstrate that our proposed SeMi outperforms existing state-of-the-art methods on multiple benchmarks, especially in reversed scenarios, where our best result shows approximately a 54.8% improvement over the baseline methods. Our code is available at https://github.com/pywin/SeMi.

Recent years have witnessed the rapid development of large language models (LLMs). Multi-modal LLMs (MLLMs) extend modality from text to various domains, attracting widespread attention due to their diverse application scenarios. As LLMs and MLLMs rely on vast amounts of model parameters and data to achieve emergent capabilities, the importance of data is gaining increasing recognition. Reviewing recent data-driven works for MLLMs, we find that the development of models and data is not two separate paths but rather interconnected. Vaster and higher-quality data improve MLLM performance, while MLLMs, in turn, facilitate the development of data. The co-development of multi-modal data and MLLMs requires a clear view of 1) at which development stages of MLLMs specific data-centric approaches can be employed to enhance certain MLLM capabilities, and 2) how MLLMs, using these capabilities, can contribute to multi-modal data in specific roles. To promote data-model co-development for MLLM communities, we systematically review existing works on MLLMs from the data-model co-development perspective.

The Intelligent Transportation System (ITS) serves as a pivotal element within urban networks, offering decision support to users and connected automated vehicles through comprehensive information gathering, sensing, device control, and data processing. Presently, ITS predominantly relies on sensors embedded in fixed infrastructure, notably Roadside Units (RSUs). However, RSUs are confined by coverage limitations and may encounter challenges in prompt emergency responses. On-demand resources, such as drones, present a viable option to supplement these deficiencies effectively. This article introduces an approach where Software-Defined Networking and Mobile Edge Computing technologies are integrated to formulate a high-availability drone swarm control and communication infrastructure framework comprising the cloud layer, edge layer, and device layer. Drones confront limitations in flight duration attributed to battery limitations, posing a challenge in sustaining continuous monitoring of road conditions over extended periods. Effective drone scheduling stands as a promising solution to overcome these constraints. To tackle this issue, we initially utilized Graph WaveNet, a specialized graph neural network structure tailored for spatial-temporal graph modeling, for training a congestion prediction model using real-world dataset inputs. Building upon this, we further propose an algorithm for drone scheduling based on congestion prediction. Our simulation experiments using real-world data demonstrate that, compared to the baseline method, the proposed scheduling algorithm not only yielded superior scheduling gains but also mitigated drone idle rates.

With the continuous rolling-out of wireless edge cloud networks, Federated Learning (FL) has emerged as a promising solution for decentralized model training without exposing raw data. However, conventional centralized FL faces several limitations in resource-constrained mobile environments, including limited privacy-preserving capabilities and substantial communication overhead, which can lead to privacy leakage. Moreover, in non-independent and identically distributed (Non IID) data environments, FL faces the critical challenge of “client drift”, which leads to performance degradation. To address these challenges, this paper proposes TWHFL, a two-stage hierarchical knowledge distillation framework for Non-IID federated learning, designed to enhance terminal privacy protection and improve model personalization under heterogeneous data distributions. Specifically, in the cloud-edge collaboration stage, edge servers generate pseudo “hard samples” for all sub-MEC centers by optimizing noise inputs guided by feature distribution statistics (e.g., batch normalization running means and variances). To alleviate label distribution skew, both the label proportions and the volume of pseudo data are dynamically adapted based on the real-time operational state of each sub-MEC center. In the edge-terminal collaboration stage, each sub-MEC center conducts localized training using both real and synthetic data without external communication, thereby significantly reducing the risk of privacy leakage. Furthermore, a joint optimization problem is formulated to determine optimal configurations of pruning rates, CPU frequencies, up-link power, and bandwidth allocation, while jointly considering constraints on convergence rate, energy consumption, and latency. Experimental results show that the proposed TWHFL framework can effectively balance privacy protection and model performance in Non-IID settings.

Existing approaches defend against backdoor attacks in federated learning (FL) mainly through a) mitigating the impact of infected models, or b) excluding infected models. The former negatively impacts model accuracy, while the latter usually relies on globally clear boundaries between benign and infected model updates. However, in reality, model updates can easily become mixed and scattered throughout due to the diverse distributions of local data. This work focuses on excluding infected models in FL. Unlike previous perspectives from a global view, we propose Snowball, a novel anti-backdoor FL framework through bidirectional elections from an individual perspective inspired by one principle deduced by us and two principles in FL and deep learning. It is characterized by a) bottom-up election, where each candidate model update votes to several peer ones such that a few model updates are elected as selectees for aggregation; and b) top-down election, where selectees progressively enlarge themselves through picking up from the candidates. We compare Snowball with state-of-the-art defenses to backdoor attacks in FL on five real-world datasets, demonstrating its superior resistance to backdoor attacks and slight impact on the accuracy of the global model.

Anomaly detection significantly enhances the robustness of cloud systems. While neural network-based methods have recently demonstrated strong advantages, they encounter practical challenges in cloud environments: the contradiction between the impracticality of maintaining a unique model for each service and the limited ability to deal with diverse normal patterns by a unified model, as well as issues with handling heavy traffic in real time and short-term anomaly detection sensitivity. Thus, we propose MACE, a multi-normal-pattern accommodated and efficient anomaly detection method in the frequency domain for time series anomaly detection. There are three novel characteristics of it: (i) a pattern extraction mechanism excelling at handling diverse normal patterns with a unified model, which enables the model to identify anomalies by examining the correlation between the data sample and its service normal pattern, instead of solely focusing on the data sample itself; (ii) a dualistic convolution mechanism that amplifies short-term anomalies in the time domain and hinders the reconstruction of anomalies in the frequency domain, which enlarges the reconstruction error disparity between anomaly and normality and facilitates anomaly detection; (iii) leveraging the sparsity and parallelism of frequency domain to enhance model efficiency. We theoretically and experimentally prove that using a strategically selected subset of Fourier bases can not only reduce computational overhead but is also profitable to distinguish anomalies, compared to using the complete spectrum. Moreover, extensive experiments demonstrate MACE's effectiveness in handling diverse normal patterns with a unified model and it achieves state-of-the-art performance with high efficiency.

Pre-trained large language models (LLMs) need fine-tuning to improve their responsiveness to natural language instructions. Federated learning offers a way to fine-tune LLMs using the abundant data on end devices without compromising data privacy. Most existing federated fine-tuning methods for LLMs rely on parameter-efficient fine-tuning techniques, which may not reach the performance height possible with full-parameter tuning. However, federated full-parameter tuning of LLMs is a non-trivial problem due to the immense communication cost. This work introduces FedKSeed that employs zeroth-order optimization with a finite set of random seeds. It significantly reduces transmission requirements between the server and clients to just a few random seeds and scalar gradients, amounting to only a few thousand bytes, making federated full-parameter tuning of billion-sized LLMs possible on devices. Building on it, we develop a strategy enabling probability-differentiated seed sampling, prioritizing perturbations with greater impact on model accuracy. Experiments across six scenarios with various LLMs, datasets and data partitions demonstrate that our approach outperforms existing federated LLM fine-tuning methods in both communication efficiency and zero-shot generalization.

Most of current anomaly detection models assume that the normal pattern remains the same all the time. However, the normal patterns of web services can change dramatically and frequently over time. The model trained on old-distribution data becomes outdated and ineffective after such changes. Retraining the whole model whenever the pattern is changed is computationally expensive. Further, at the beginning of normal pattern changes, there is not enough observation data from the new distribution. Retraining a large neural network model with limited data is vulnerable to overfitting. Thus, we propose a Light Anti-overfitting Retraining Approach (LARA) based on deep variational auto-encoders for time series anomaly detection. In LARA we make the following three major contributions: 1) the retraining process is designed as a convex problem such that overfitting is prevented and the retraining process can converge fast; 2) a novel ruminate block is introduced, which can leverage the historical data without the need to store them; 3) we mathematically and experimentally prove that when fine-tuning the latent vector and reconstructed data, the linear formations can achieve the least adjusting errors between the ground truths and the fine-tuned ones. Moreover, we have performed many experiments to verify that retraining LARA with even a limited amount of data from new distribution can achieve competitive performance in comparison with the state-of-the-art anomaly detection models trained with sufficient data. Besides, we verify its light computational overhead.

In cross-silo federated learning (FL), the data among clients are usually statistically heterogeneous (aka not independent and identically distributed, non-IID) due to diversified data sources, lowering the accuracy of FL. Although many personalized FL (PFL) approaches have been proposed to address this issue, they are only suitable for data with specific degrees of statistical heterogeneity. In the real world, the heterogeneity of data among clients is often immeasurable due to privacy concern, making the targeted selection of PFL approaches difficult. Besides, in cross-silo FL, clients are usually from different organizations, tending to hold architecturally different private models. In this work, we propose a novel FL framework, FedAPEN, which combines mutual learning and ensemble learning to take the advantages of private and shared global models while allowing heterogeneous models. Within FedAPEN, we propose two mechanisms to coordinate and promote model ensemble such that FedAPEN achieves excellent accuracy on various data distributions without prior knowledge of data heterogeneity, and thus, obtains the adaptability to data heterogeneity. We conduct extensive experiments on four real-world datasets, including: 1) Fashion MNIST, CIFAR-10, and CIFAR-100, each with ten different types and degrees of label distribution skew; and 2) eICU with feature distribution skew. The experiments demonstrate that FedAPEN almost obtains superior accuracy on data with varying types and degrees of heterogeneity compared with baselines.

Recently, edge user allocation (EUA) problem has received much attentions. It aims to appropriately allocate edge users to their nearby edge servers. Existing EUA approaches suffer from a series of limitations. First, considering users' service requests only as a whole, they neglect the fact that a service request may be partitioned into multiple tasks to be performed by different servers. Second, the impact of the spatial distance between edge users and servers on users' quality of experience is not properly considered. Third, the temporal dynamics of users' service requests has not been fully investigated. To overcome these limitations systematically, this paper focuses on the problem of spatio-temporal edge user allocation with task decomposition (ST-EUA). We first formulate the ST-EUA problem. Then, we transform ST-EUA problem as a multi-objective optimization problem and prove its NP-hardness. To tackle the ST-EUA problem effectively and efficiently, we propose a novel genetic algorithm-based heuristic approach GA-ST, aiming to maximize usersoverall QoE while minimizing migration cost in different time slots. Extensive experiments are conducted on two widely-used real-world datasets to evaluate the performance of GA-ST. The results demonstrate that GA-ST significantly outperforms state-of-the-art approaches in finding approximate solutions in terms of the trade-off among multiple metrics.

With an growing number of web services available on the Internet, an increasing burden is imposed on the use and management of service repository. Service clustering has been employed to facilitate a wide range of service-oriented tasks, such as service discovery, selection, composition and recommendation. Conventional approaches have been proposed to cluster web services by using explicit features, including syntactic features contained in service descriptions or semantic features extracted by probabilistic topic models. However, service implicit features are ignored and have yet to be properly explored and leveraged. To this end, we propose a novel heuristics-based framework DeepWSC for web service clustering. It integrates deep semantic features extracted from service descriptions by an improved recurrent convolutional neural network and service composability features obtained from service invocation relationships by a signed graph convolutional network, to jointly generate integrated implicit features for web service clustering. Extensive experiments are conducted on 8,459 real-world web services. The experiment results demonstrate that DeepWSC outperforms state-of-the-art approaches for web service clustering in terms of multiple evaluation metrics.

Correlative approaches have attempted to cluster web services based on either the explicit information contained in service descriptions or functionality semantic features extracted by probabilistic topic models. However, the implicit contextual information of service descriptions is ignored and has yet to be properly explored and leveraged. To this end, we propose a novel framework with deep neural network, called DeepWSC, which combines the advantages of recurrent neural network and convolutional neural network to cluster web services through automatic feature extraction. The experimental results demonstrate that DeepWSC outperforms state-of-the-art approaches for web service clustering in terms of multiple evaluation metrics.