Fine-tuning large language models (LLMs) in federated settings enables privacy-preserving adaptation but suffers from cross-client interference due to model aggregation. Existing federated LoRA fine-tuning methods, primarily based on FedAvg, struggle with data heterogeneity, leading to harmful cross-client interference and suboptimal personalization. In this work, we propose FedALT, a novel personalized federated LoRA fine-tuning algorithm that fundamentally departs from FedAvg. Instead of using an aggregated model to initialize local training, each client continues training its individual LoRA while incorporating shared knowledge through a separate Rest-of-World (RoW) LoRA component. To effectively balance local adaptation and global information, FedALT introduces an adaptive mixer that dynamically learns input-specific weightings between the individual and RoW LoRA components, drawing conceptual foundations from the Mixture-of-Experts (MoE) paradigm. Through extensive experiments on NLP benchmarks, we demonstrate that FedALT significantly outperforms state-of-the-art personalized federated LoRA fine-tuning methods, achieving superior local adaptation without sacrificing computational efficiency.

Foundation models (FMs) achieve strong performance across diverse tasks with task-specific fine-tuning, yet full parameter fine-tuning is often computationally prohibitive for large models. Parameter-efficient fine-tuning (PEFT) methods like Low-Rank Adaptation (LoRA) reduce this cost by introducing low-rank matrices for tuning fewer parameters. While LoRA allows for efficient fine-tuning, it requires significant data for adaptation, making Federated Learning (FL) an appealing solution due to its privacy-preserving collaborative framework. However, combining LoRA with FL introduces two key challenges: the \textbf{Server-Side Aggregation Bias}, where server-side averaging of LoRA matrices diverges from the ideal global update, and the \textbf{Client-Side Initialization Lag}, emphasizing the need for consistent initialization across rounds. Existing approaches address these challenges individually, limiting their effectiveness. We propose LoRA-FAIR, a novel method that tackles both issues by introducing a correction term on the server, enhancing aggregation efficiency and accuracy. LoRA-FAIR maintains computational and communication efficiency, yielding superior performance over state-of-the-art methods. Experimental results on ViT and MLP-Mixer models across large-scale datasets demonstrate that LoRA-FAIR consistently achieves performance improvements in FL settings.

Large Language Models (LLMs) have demonstrated impressive capabilities across various tasks, but fine-tuning them for domain-specific applications often requires substantial domain-specific data that may be distributed across multiple organizations. Federated Learning (FL) offers a privacy-preserving solution, but faces challenges with computational constraints when applied to LLMs. Low-Rank Adaptation (LoRA) has emerged as a parameter-efficient fine-tuning approach, though a single LoRA module often struggles with heterogeneous data across diverse domains. This paper addresses two critical challenges in federated LoRA fine-tuning: 1. determining the optimal number and allocation of LoRA experts across heterogeneous clients, and 2. enabling clients to selectively utilize these experts based on their specific data characteristics. We propose FedLEASE (Federated adaptive LoRA Expert Allocation and SElection), a novel framework that adaptively clusters clients based on representation similarity to allocate and train domain-specific LoRA experts. It also introduces an adaptive top-$M$ Mixture-of-Experts mechanism that allows each client to select the optimal number of utilized experts. Our extensive experiments on diverse benchmark datasets demonstrate that FedLEASE significantly outperforms existing federated fine-tuning approaches in heterogeneous client settings while maintaining communication efficiency.

Federated learning (FL) enables distributed devices to collaboratively train machine learning (ML) models while maintaining data privacy. However, the heterogeneous hardware capabilities of participating devices often result in significant training delays, as straggler clients with limited resources prolong the aggregation process. Existing solutions such as client selection, asynchronous FL, and partial training partially address these challenges but encounter issues such as reduced accuracy, stale updates, and compromised model performance due to inconsistent training contributions. To overcome these limitations, we propose FedEL, a federated elastic learning framework that enhances training efficiency while maintaining model accuracy. FedEL introduces a novel window-based training process, sliding the window to locate the training part of the model and dynamically selecting important tensors for training within a coordinated runtime budget. This approach ensures progressive and balanced training across all clients, including stragglers. Additionally, FedEL employs a tensor importance adjustment module, harmonizing local and global tensor importance to mitigate biases caused by data heterogeneity. The experiment results shows that FedEL achieves up to 3.87× improvement in time-to-accuracy compared to baselines while maintaining or exceeding final test accuracy.

Federated Learning (FL) is a distributed machine learning framework that efficiently reduces communication and preserves privacy. Existing FL algorithms typically rely on the assumption of direct communication between the server and clients for model data exchange. However, this assumption does not apply in many real-world scenarios where appropriate communication infrastructure is lacking, such as in remote smart sensing. To overcome this challenge, we propose a new framework, FedEx (Federated Learning via Model Express Delivery). FedEx employs mobile transporters, such as Unmanned Aerial Vehicles (UAVs), to establish indirect communication channels between the server and clients. We have developed two algorithms under this framework: FedEx-Sync and FedEx-Async, which differ based on whether the transporters operate on a synchronized or asynchronized schedule. Although indirect communication introduces variable delays in global model dissemination and local model collection, we demonstrate the convergence of both FedEx versions. Additionally, we explore the energy consumption of transporters, integrating it with the convergence bounds and proposing a bi-level optimization algorithm for efficient client assignment and route planning. Our experiments, conducted on two public datasets in a simulated environment, further demonstrate the efficacy of FedEx.

The fusion of complementary multimodal information is crucial in computational pathology for accurate diagnostics. However, existing multimodal learning approaches necessitate access to users’ raw data, posing substantial privacy risks. While Federated Learning (FL) serves as a privacy-preserving alternative, it falls short in addressing the challenges posed by heterogeneous (yet possibly overlapped) modalities data across various hospitals. To bridge this gap, we propose a Federated Multi-Modal (FedMM) learning framework that federatedly trains multiple single-modal feature extractors to enhance subsequent classification performance instead of existing FL that aims to train a unified multimodal fusion model. Any participating hospital, even with small-scale datasets or limited devices, can leverage these federated trained extractors to perform local downstream tasks (e.g., classification) while ensuring data privacy. Through comprehensive evaluations of two publicly available datasets, we demonstrate that FedMM notably outperforms two baselines in accuracy and AUC metrics.

Federated learning (FL) with noisy labels poses a significant challenge. Existing methods designed for handling noisy labels in centralized learning tend to lose their effectiveness in the FL setting, mainly due to the small dataset size and the heterogeneity of client data. While some attempts have been made to tackle FL with noisy labels, they primarily focused on scenarios involving class-conditional noise. In this paper, we study the more challenging and practical issue of instance-dependent noise (IDN) in FL. We introduce a novel algorithm called FedBeat (Federated Learning with Bayesian Ensemble-Assisted Transition Matrix Estimation). FedBeat aims to build a global statistically consistent classifier using the IDN transition matrix (IDNTM), which encompasses three synergistic steps: (1) A federated data extraction step that constructs a weak global model and extracts high-confidence data using a Bayesian model ensemble method. (2) A federated transition matrix estimation step in which clients collaboratively train an IDNTM estimation network based on the extracted data. (3) A federated classifier correction step that enhances the global model’s performance by training it using a loss function tailored for noisy labels, leveraging the IDNTM. Experiments conducted on CIFAR-10 and SVHN verify that the proposed method significantly outperforms state-of-the-art methods.

Distributed quantum computing (DQC) holds immense promise in harnessing the potential of quantum computing by interconnecting multiple small quantum computers (QCs) through a quantum data network (QDN). Establishing long-distance quantum entanglement between two QCs for quantum teleportation within the QDN is a critical aspect, and it involves entanglement routing - finding a route between QCs and efficiently allocating qubits along that route. Existing approaches have mainly focused on optimizing entanglement performance for current entanglement connection (EC) requests. However, they often overlook the user's perspective, wherein the user making EC requests operates under a budget constraint over an extended period. Furthermore, both QDN resources (quantum channels and qubits) and the EC requests, reflecting the DQC workload, vary over time. In this paper, we present a novel user-centric entanglement routing problem that spans an extended period to maximize the entanglement success rate while adhering to the user's budget constraint. To address this challenge, we leverage the Lyapunov drift-plus-penalty framework to decompose the long-term optimization problem into per-slot problems, allowing us to find solutions using only the current system information. Subsequently, we develop efficient algorithms based on continuous-relaxation and Gibbs-sampling techniques to solve the per-slot entanglement routing problem. Theoretical performance guarantees are provided for both the per-slot and long-term problems. Extensive simulations demonstrate that our algorithm significantly outperforms baseline approaches in terms of entanglement success rate and budget adherence.

Federated learning (FL) allows collaborative machine learning training without sharing private data. While most FL methods assume identical data domains across clients, real-world scenarios often involve heterogeneous data domains. Federated Prototype Learning (FedPL) addresses this issue, using mean feature vectors as prototypes to enhance model generalization. However, existing FedPL methods create the same number of prototypes for each client, leading to cross-domain performance gaps and disparities for clients with varied data distributions. To mitigate cross-domain feature representation variance, we introduce FedPLVM, which establishes variance-aware dual-level prototypes clustering and employs a novel $\alpha$-sparsity prototype loss. The dual-level prototypes clustering strategy creates local clustered prototypes based on private data features, then performs global prototypes clustering to reduce communication complexity and preserve local data privacy. The $\alpha$-sparsity prototype loss aligns samples from underrepresented domains, enhancing intra-class similarity and reducing inter-class similarity. Evaluations on Digit-5, Office-10, and DomainNet datasets demonstrate our method's superiority over existing approaches.

Multimodal federated learning (FL) targets the intersection of two promising research directions in Internet of Things (IoT) scenarios: 1) leveraging complementary multimodal information to enhance downstream inference performance and 2) conducting distributed training with privacy protection. However, the majority of existing works primarily focus on applying different FL methods in a straightforward manner after the multimodal feature fusion stage without fundamentally disentangling the multimodal FL across both the feature space and the sample space. There still exists an important tradeoff between the computationally demanding nature of multimodal information and the limited computing resources in IoT systems. To tackle this challenge, we propose a hybrid FL algorithm tailored for multimodal IoT systems (HFM). HFM utilizes vertical FL (VFL) to distribute computing resources across the feature space and horizontal FL (HFL) to distribute computing resources across the sample space. This innovative algorithm necessitates consideration of both stale information from the VFL component and perturbed gradients from the HFL component, which is not fully understood from a theoretical point. In this article, we theoretically prove that the convergence of HFM depends on the frequency of VFL communication and HFL communication, as well as the number of vertical partitions and horizontal partitions. Furthermore, we empirically demonstrate that HFM outperforms three types of baselines based on two public multimodal data sets, thereby making it practical for multimodal IoT systems that require rapid and accurate downstream inference tasks, such as classification, prediction, etc.

In this study, we address the emerging field of streaming federated learning (SFL) and propose local cache update rules to manage dynamic data distributions and limited cache capacity. Traditional federated learning (FL) relies on fixed data sets, whereas in SFL, data is streamed, and its distribution changes over time, leading to discrepancies between the local training data set and long-term distribution. To mitigate this problem, we propose three local cache update rules—first-in–first-out (FIFO), static ratio selective replacement (SRSR), and dynamic ratio selective replacement (DRSR)—that update the local cache of each client while considering the limited cache capacity. Furthermore, we derive a convergence bound for our proposed SFL algorithm as a function of the distribution discrepancy between the long-term data distribution and the client’s local training data set. We then evaluate our proposed algorithm on two data sets: 1) a network traffic classification data set and 2) an image classification data set. Our experimental results demonstrate that our proposed local cache update rules significantly reduce the distribution discrepancy and outperform the baseline methods. Our study advances the field of SFL and provides practical cache management solutions in FL.