Different from existing federated fine-tuning (FFT) methods for foundation models, hybrid heterogeneous federated fine-tuning (HHFFT) is an under-explored scenario where clients exhibit double heterogeneity in model architectures and downstream tasks. This hybrid heterogeneity introduces two significant challenges: 1) heterogeneous matrix aggregation, where clients adopt different large-scale foundation models based on their task requirements and resource limitations, leading to dimensional mismatches during LoRA parameter aggregation; and 2) multi-task knowledge interference, where local shared parameters, trained with both task-shared and task-specific knowledge, cannot ensure only task-shared knowledge is transferred between clients. To address these challenges, we propose H2Tune, a federated foundation model fine-tuning with hybrid heterogeneity. Our framework H2Tune consists of three key components: (i) sparsified triple matrix decomposition to align hidden dimensions across clients through constructing rank-consistent middle matrices, with adaptive sparsification based on client resources; (ii) relation-guided matrix layer alignment to handle heterogeneous layer structures and representation capabilities; and (iii) alternating task-knowledge disentanglement mechanism to decouple shared and specific knowledge of local model parameters through alternating optimization. Theoretical analysis proves a convergence rate of O(1/√T). Extensive experiments show our method achieves up to 15.4% accuracy improvement compared to state-of-the-art baselines.

Different from existing vertical federated learning (VFL), zero-shot VFL (ZVFL) is an under-explored scenario where test classes are absent from partial parties' training sets. In extreme cases, some test classes even have no training samples for all parties. Traditionally, existing zero-shot methods require abundant seen-class samples for effective knowledge transfer to recognize unseen classes. However, both the limited aligned samples and different seen classes pose several unique challenges to ZVFL. The primary challenge lies in the seen-to-unseen transfer insufficiency, as the scarcity of aligned samples and diverse seen-class distributions across parties severely limits the model's capability to learn discriminative features that can generalize to unseen classes. Moreover, the multi-party bias inconsistency arises as different parties tend to be biased towards their own seen classes during prediction, leading to skewed classification results at the active party. To address these challenges, we propose FEZE, an alignment-flexible zero-shot vertical federated learning framework. Specifically, we introduce a relation learning network to capture class-feature relationships between class labels and feature representations across heterogeneous feature spaces, enabling unseen class recognition through relationship inference. Additionally, we design a meta-relation learning mechanism that leverages diverse class-feature patterns to tackle the insufficient feature generalization from limited seen-class samples. Finally, we propose an alignment-flexible adaptive aggregation strategy that achieves adaptively aggregation based on inconsistent prediction spaces with arbitrary number of aligned samples. Theoretical analysis proves that FEZE can achieve a convergence rate of O(1/T). In the most challenging zero-shot scenario without aligned samples, FEZE surpasses state-of-the-art baselines by an average of 7.47% across three datasets.

Conversational Recommender Systems (CRS) have recently attracted significant attention. Despite existing GNN-based CRS methods have been proven to be effective in exploiting knowledge graphs (KGs), we note that these methods are not suitable for modeling scenarios with geographic positional information, which encompass the two key issues that have not been adequately solved: 1) Data noise is ubiquitous in the real world due to a variety of factors, and existing methods are prone to amplifying data noise, which can lead to a deterioration in downstream tasks; 2) Existing CRS models are designed solely in Euclidean space without considering space curvature, which implies that they may suffer from significant distortion when representing real-world graph structures, leading to a decrease in the accuracy and reliability of geographical POIs. To this end, we propose a Light POI-Guided Conversational Recommender based on Adaptive Space, namely PCRA, aiming to address the above problems by enhancing both embedding spaces and graph structures. Specifically, PCRA introduces the unified space to obtain high-quality embeddings compatible with hyperbolic space, Euclidean space, and spherical space. On the other hand, to extract the most valuable neighbors, we adopt a graph denoising module to eliminate noisy entities and ensure light information propagation. Finally, we further fuse the embeddings of utterances and entities to bridge the semantic gap of recommendation and conversation. Extensive experiments on MultiWOZ 2.0 and MultiWOZ 2.1 datasets demonstrate that our proposed PCRA has a significant improvement over the state-of-the-art CRS methods.

Abstract Federated learning (FL) is a novel distributed machine learning paradigm that enables participants to collaboratively train a centralized model with privacy preservation by eliminating the requirement of data sharing. In practice, FL often involves multiple participants and requires the third party to aggregate global information to guide the update of the target participant. Therefore, many FL methods do not work well due to the training and test data of each participant may not be sampled from the same feature space and the same underlying distribution. Meanwhile, the differences in their local devices (system heterogeneity), the continuous influx of online data (incremental data), and labeled data scarcity may further influence the performance of these methods. To solve this problem, federated transfer learning (FTL), which integrates transfer learning (TL) into FL, has attracted the attention of numerous researchers. However, since FL enables a continuous share of knowledge among participants with each communication round while not allowing local data to be accessed by other participants, FTL faces many unique challenges that are not present in TL. In this survey, we focus on categorizing and reviewing the current progress on federated transfer learning, and outlining corresponding solutions and applications. Furthermore, the common setting of FTL scenarios, available datasets, and significant related research are summarized in this survey.

The prior knowledge, such as expert rules and knowledge base, has been proven effective in the traditional Biomedical Named Entity Recognition (BioNER). Most current neural BioNER systems use this external knowledge for pre-processing or post-editing instead of incorporate it into the training process, which cannot be learned by the model. To encode prior knowledge into the model, we present a unified multi-task Machine Reading Comprehension (MRC) framework for BioNER. Specifically, in the MRC task, the question sequences are derived from the standard BioNER dataset. We introduce three kinds of prior knowledge at query sequences, including Wikipedia, annotation scheme, entity dictionary. Then, our model adopts a multi-task learning strategy to joint training the main task BioNER and the auxiliary task MRC. Finally, experimental results on three benchmark datasets validate the superiority of our BioNER model compared with various state-of-the-art baselines.

MOTIVATION: Biomedical Named Entity Recognition (BioNER) aims to identify biomedical domain-specific entities (e.g. gene, chemical and disease) from unstructured texts. Despite deep learning-based methods for BioNER achieving satisfactory results, there is still much room for improvement. Firstly, most existing methods use independent sentences as training units and ignore inter-sentence context, which usually leads to the labeling inconsistency problem. Secondly, previous document-level BioNER works have approved that the inter-sentence information is essential, but what information should be regarded as context remains ambiguous. Moreover, there are still few pre-training-based BioNER models that have introduced inter-sentence information. Hence, we propose a cache-based inter-sentence model called BioNER-Cache to alleviate the aforementioned problems. RESULTS: We propose a simple but effective dynamic caching module to capture inter-sentence information for BioNER. Specifically, the cache stores recent hidden representations constrained by predefined caching rules. And the model uses a query-and-read mechanism to retrieve similar historical records from the cache as the local context. Then, an attention-based gated network is adopted to generate context-related features with BioBERT. To dynamically update the cache, we design a scoring function and implement a multi-task approach to jointly train our model. We build a comprehensive benchmark on four biomedical datasets to evaluate the model performance fairly. Finally, extensive experiments clearly validate the superiority of our proposed BioNER-Cache compared with various state-of-the-art intra-sentence and inter-sentence baselines. AVAILABILITYAND IMPLEMENTATION: Code will be available at https://github.com/zgzjdx/BioNER-Cache. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Research on document-level Neural Machine Translation (NMT) models has attracted increasing attention in recent years.Although the proposed works have proved that the inter-sentence information is helpful for improving the performance of the NMT models, what information should be regarded as context remains ambiguous.To solve this problem, we proposed a novel cache-based document-level NMT model which conducts dynamic caching guided by theme-rheme information.The experiments on NIST evaluation sets demonstrate that our proposed model achieves substantial improvements over the state-of-the-art baseline NMT models.As far as we know, we are the first to introduce theme-rheme theory into the field of machine translation.