Federated learning (FL) has emerged as a promising distributed learning paradigm for training deep neural networks (DNNs) at the wireless edge, but its performance can be severely hindered by unreliable wireless transmission and inherent data heterogeneity among clients. Existing solutions primarily address these challenges by incorporating wireless resource optimization strategies, often focusing on uplink resource allocation across clients under the assumption of homogeneous client-server network standards. However, these approaches overlooked the fact that mobile clients may connect to the server via diverse network standards (e.g., 4G, 5G, Wi-Fi) with customized configurations, limiting the flexibility of server-side modifications and restricting applicability in real-world commercial networks. This paper presents a novel theoretical analysis about how transmission failures in unreliable networks distort the effective label distributions of local samples, causing deviations from the global data distribution and introducing convergence bias in FL. Our analysis reveals that a carefully designed client selection strategy can mitigate biases induced by network unreliability and data heterogeneity. Motivated by this insight, we propose FedCote, a client selection approach that optimizes client selection probabilities without relying on wireless resource scheduling. Experimental results demonstrate the robustness of FedCote in DNN-based classification tasks under unreliable networks with frequent transmission failures.

Zhigen Li, Jianxiang Peng, Yanmeng Wang, Yong Cao, Tianhao Shen, Minghui Zhang, Linxi Su, Shang Wu, Yihang Wu, YuQian Wang, Ye Wang, Wei Hu, Jianfeng Li, Shaojun Wang, Jing Xiao, Deyi Xiong. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Aligning Large Language Models (LLMs) with general human preferences has been proved crucial in improving the interaction quality between LLMs and human. However, human values are inherently diverse among different individuals, making it insufficient to align LLMs solely with general preferences. To address this, personalizing LLMs according to individual feedback emerges as a promising solution. Nonetheless, this approach presents challenges in terms of the efficiency of alignment algorithms. In this work, we introduce a flexible paradigm for individual preference alignment. Our method fundamentally improves efficiency by disentangling preference representation from text generation in LLMs. We validate our approach across multiple text generation tasks and demonstrate that it can produce aligned quality as well as or better than PEFT-based methods, while reducing additional training time for each new individual preference by 80% to 90% in comparison with them.

Modern machine learning applications are characterized by the increasing size of deep models and the growing diversity of data modalities. This trend underscores the importance of efficiently adapting pre-trained multi-modal models to the test distribution in real time, i.e., multi-modal test-time adaptation. In practice, the magnitudes of multi-modal shifts vary because multiple data sources interact with the impact factor in diverse manners. In this research, we investigate the the under-explored practical scenario uni-modal distribution shift, where the distribution shift influences only one modality, leaving the others unchanged. Through theoretical and empirical analyses, we demonstrate that the presence of such shift impedes multi-modal fusion and leads to the negative transfer phenomenon in existing test-time adaptation techniques. To flexibly combat this unique shift, we propose a selective adaptation schema that incorporates multiple modality-specific adapters to accommodate potential shifts and a “router” module that determines which modality requires adaptation. Finally, we validate the effectiveness of our proposed method through extensive experimental evaluations. Code available at https://github.com/chenmc1996/Uni-Modal-Distribution-Shift.

As a promising distributed learning paradigm, federated learning (FL) involves training deep neural network (DNN) models at the network edge while protecting the privacy of the edge clients. To train a large-scale DNN model, batch normalization (BN) has been regarded as a simple and effective means to accelerate the training and improve the generalization capability. However, recent findings indicate that BN can significantly impair the performance of FL in the presence of non-i.i.d. data. While several FL algorithms have been proposed to address this issue, their performance still falls significantly when compared to the centralized scheme. Furthermore, none of them have provided a theoretical explanation of how the BN damages the FL convergence. In this article, we present the first convergence analysis to show that under the non-i.i.d. data, the mismatch between the local and global statistical parameters in BN causes the gradient deviation between the local and global models, which, as a result, slows down and biases the FL convergence. In view of this, we develop a new FL algorithm that is tailored to BN, called FedTAN, which is capable of achieving robust FL performance under a variety of data distributions via iterative layer-wise parameter aggregation. Comprehensive experimental results demonstrate the superiority of the proposed FedTAN over existing baselines for training BN-based DNN models.

Federated Learning (FL) is a promising privacy-preserving distributed learning paradigm but suffers from high communi- cation cost when training large-scale machine learning models. Sign-based methods, such as SignSGD, have been proposed as a biased gradient compression technique for reducing the communication cost. However, sign-based algorithms could diverge under heterogeneous data, which thus motivated the de- velopment of advanced techniques, such as the error-feedback method and stochastic sign-based compression, to fix this issue. Nevertheless, these methods still suffer from slower convergence rates, and none of them allows multiple local SGD updates like FedAvg. In this paper, we propose a novel noisy perturbation scheme with a general symmetric noise distribution for sign-based compression, which not only al- lows one to flexibly control the bias-variance tradeoff for the compressed gradient, but also provides a unified viewpoint to existing stochastic sign-based methods. More importantly, the proposed scheme enables the development of the very first sign-based FedAvg algorithm (z-SignFedAvg) to accelerate the convergence. Theoretically, we show that z-SignFedAvg achieves a faster convergence rate than existing sign-based methods and, under the uniformly distributed noise, can enjoy the same convergence rate as its uncompressed counterpart. Extensive experiments are conducted to demonstrate that the z-SignFedAvg can achieve competitive empirical performance on real datasets and outperforms existing schemes.

Identifying sentences sharing similar meanings is crucial to speech and text understandings. Although currently popular cross-encoder solutions with pre-trained language models as backbone have achieved remarkable performance, they suffer from the lack of the permutation invariance or symmetry that is one of the most important inductive biases to such task. To alleviate this issue, in this research we propose a permutation invariant training framework, in which a symmetry regularization is introduced during training that forces the model to produce the same predictions for input sentence pairs in both forward and backward directions. Empirical studies exhibit improved performance over competitive baselines.

Batch normalization (BN) has been widely used for accelerating the training of deep neural networks. However, recent findings show that, in the federated learning (FL) scenarios, BN can damage the learning performance when the clients have non-i.i.d. data. While several FL schemes have been proposed to address this issue, they still suffer a significant performance loss compared to the centralized scheme. In addition, none of them have explained how the BN impacts the FL convergence analytically. In this paper, we present the first convergence analysis to show that the mismatched local and global statistical parameters due to non-i.i.d data cause gradient deviation and it leads the algorithm to converge to a biased solution with a slower rate. To remedy this, we further present a new FL algorithm, called FedTAN, based on an iterative layer-wise parameter aggregation procedure. Experiment results are presented to show the superiority of FedTAN.

Retrieval Question Answering (ReQA) is an essential mechanism of information sharing which aims to find the answer to a posed question from large-scale candidates. Currently, the most efficient solution is Dual-Encoder which has shown great potential in the general domain, while it still lacks research on biomedical ReQA. Obtaining a robust Dual-Encoder from biomedical datasets is challenging, as scarce annotated data are not enough to sufficiently train the model which results in over-fitting problems. In this work, we first build ReQA BioASQ datasets for retrieving answers to biomedical questions, which can facilitate the corresponding research. On that basis, we propose a framework to solve the over-fitting issue for robust biomedical answer retrieval. Under the proposed framework, we first pre-train Dual-Encoder on natural language inference (NLI) task before the training on biomedical ReQA, where we appropriately change the pre-training objective of NLI to improve the consistency between NLI and biomedical ReQA, which significantly improve the transferability. Moreover, to eliminate the feature redundancies of Dual-Encoder, consistent post-whitening is proposed to conduct decorrelation on the training and trained sentence embeddings. With extensive experiments, the proposed framework achieves promising results and exhibits significant improvement compared with various competitive methods.

Biomedical factoid question answering is an essential application for biomedical information sharing. Recently, neural network based approaches have shown remarkable performance for this task. However, due to the scarcity of annotated data which requires intensive knowledge of expertise, training a robust model on limited-scale biomedical datasets remains a challenge. Previous works solve this problem by introducing useful knowledge. It is found that the interaction between question and answer (QA-interaction) is also a kind of knowledge which could help extract answer accurately. This research develops a knowledge distillation framework for biomedical factoid question answering, in which a teacher model as the knowledge source of QA-interaction is designed to enhance the student model. In addition, to further alleviate the problem of limited-scale dataset, a novel adversarial knowledge distillation technique is proposed to robustly distill the knowledge from teacher model to student model by constructing perturbed examples as additional training data. By forcing the student model to mimic the predicted distributions of teacher model on both original examples and perturbed examples, the knowledge of QA-interaction can be learned by student model. We evaluate the proposed framework on the widely used BioASQ datasets, and experimental results have shown the proposed method's promising potential.

Paraphrase generation is a longstanding NLP task and achieves great success with the aid of large corpora.However, transferring a paraphrasing model to another domain encounters the problem of domain shifting especially when the data is sparse.At the same time, widely using large pre-trained language models (PLMs) faces the overfitting problem when training on scarce labeled data.To mitigate these two issues, we propose, LAPA, an effective adapter for PLMs optimized by meta-learning.LAPA has three-stage training on three types of related resources to solve this problem: 1. pre-training PLMs on unsupervised corpora, 2. inserting an adapter layer and meta-training on source domain labeled data, and 3. fine-tuning adapters on a small amount of target domain labeled data.This method enables paraphrase generation models to learn basic language knowledge first, then learn the paraphrasing task itself later, and finally adapt to the target task.Our experimental results demonstrate that LAPA achieves state-of-the-art in supervised, unsupervised, and low-resource settings on three benchmark datasets.With only 2% of trainable parameters and 1% labeled data of the target task, our approach can achieve a competitive performance with previous work.

Variational autoencoders (VAEs) are one of the most powerful unsupervised learning frameworks in NLP for latent representation learning and latent-directed generation. The classic optimization goal of VAEs is to maximize the Evidence Lower Bound (ELBo), which consists of a conditional likelihood for generation and a negative Kullback-Leibler (KL) divergence for regularization. In practice, optimizing ELBo often leads the posterior distribution of all samples converging to the same degenerated local optimum, namely posterior collapse or KL vanishing. There are effective ways proposed to prevent posterior collapse in VAEs, but we observe that they in essence make trade-offs between posterior collapse and the hole problem, i.e., the mismatch between the aggregated posterior distribution and the prior distribution. To this end, we introduce new training objectives to tackle both problems through a novel regularization based on the probabilistic density gap between the aggregated posterior distribution and the prior distribution. Through experiments on language modeling, latent space visualization, and interpolation, we show that our proposed method can solve both problems effectively and thus outperforms the existing methods in latent-directed generation. To the best of our knowledge, we are the first to jointly solve the hole problem and posterior collapse.

Federated learning (FL) has been recognized as a viable distributed learning paradigm which trains a machine learning model collaboratively with massive mobile devices in the wireless edge while protecting user privacy. Although various communication schemes have been proposed to expedite the FL process, most of them have assumed ideal wireless channels which provide reliable and lossless communication links between the server and mobile clients. Unfortunately, in practical systems with limited radio resources such as constraint on the training latency and constraints on the transmission power and bandwidth, transmission of a large number of model parameters inevitably suffers from quantization errors (QE) and transmission outage (TO). In this paper, we consider such non-ideal wireless channels, and carry out the first analysis showing that the FL convergence can be severely jeopardized by TO and QE, but intriguingly can be alleviated if the clients have uniform outage probabilities. These insightful results motivate us to propose a robust FL scheme, namedFedTOE, which performs joint allocation of wireless resources and quantization bits across the clients to minimize the QE while making the clients have the same TO probability. Extensive experimental results are presented to show the superior performance ofFedTOEfor deep learning-based classification tasks with transmission latency constraints.

Proper understanding of paragraph is essential for question generation task since the semantic interaction is complicated among sentences. How to integrate long text paragraph information into question generation is still a challenge. In this research, we proposed a multi-perspective paragraph context modeling mechanism, which firstly encodes the contextualized representation of input paragraph, and then utilize multi-head self-attention and Rezero network to further enhance paragraph-level feature extraction and context modeling. Finally, attention-based decoder with copy mechanism generates question according to encoded hidden states. Experimental study on widely used SQuAD dataset has shown the proposed method’s potential.

Current generative dialogue systems tend to produce generic dialog responses, which lack useful information and semantic coherence. An promising method to alleviate this problem is to integrate knowledge triples from knowledge base. However, current approaches mainly augment Seq2Seq framework with knowledge-aware mechanism to retrieve a large number of knowledge triples without considering specific dialogue context, which probably results in knowledge redundancy and incomplete knowledge comprehension. In this paper, we propose to leverage the contextual word representation of dialog post to filter out irrelevant knowledge with an attention-based triple filter network. We introduce a novel knowledge-enriched framework to integrate the filtered knowledge into the dialogue representation. Entity copy is further proposed to facilitate the integration of the knowledge during generation. Experiments on dialogue generation tasks have shown the proposed framework’s promising potential.

Generating on-topic responses and understanding the background information of context are both significant for dialogue generation. However, few works simultaneously concentrate on these two issues. For this purpose, we propose an open-domain topic-aware dialogue generation model via joint learning. We first design two-hop based static graph attention mechanism to enhance the semantic representations of context, and then two auxiliary sub-tasks are introduced. Topic Predictor module is designed to focus on the most pertinent topics and Language Modeling module further facilitates learning richer information from context. Experimental study has shown the proposed model’s promising potential. In particular, our model predicts the most topics that best match the query per response. Besides, further analysis proves that our model can generate more diversified and informative responses.

BACKGROUND: Biomedical question answering (QA) is a sub-task of natural language processing in a specific domain, which aims to answer a question in the biomedical field based on one or more related passages and can provide people with accurate healthcare-related information. Recently, a lot of approaches based on the neural network and large scale pre-trained language model have largely improved its performance. However, considering the lexical characteristics of biomedical corpus and its small scale dataset, there is still much improvement room for biomedical QA tasks. RESULTS: Inspired by the importance of syntactic and lexical features in the biomedical corpus, we proposed a new framework to extract external features, such as part-of-speech and named-entity recognition, and fused them with the original text representation encoded by pre-trained language model, to enhance the biomedical question answering performance. Our model achieves an overall improvement of all three metrics on BioASQ 6b, 7b, and 8b factoid question answering tasks. CONCLUSIONS: The experiments on BioASQ question answering dataset demonstrated the effectiveness of our external feature-enriched framework. It is proven by the experiments conducted that external lexical and syntactic features can improve Pre-trained Language Model's performance in biomedical domain question answering task.

Teaching machine to answer arbitrary questions is a long-term goal of natural language processing. In real dialogue corpus, informative words like named entities can often be infrequent and hard to model, and one primary challenge of dialogue system is how to promote the model's capability of generating high-quality responses with those informative words. In order to address this problem, we propose a novel pre-training based encoder-decoder model, which can enhance the multiturn dialogue response generation by incorporating external textual knowledge. We adopt BERT as encoder to merge external knowledge into dialogue history modeling, and a multi-head attention based decoder is designed to incorporate the semantic information from both knowledge and dialogue hidden representations into decoding process to generate informative and proper dialogue responses. Experiments on two response generation tasks indicate our model to be superior over competitive baselines on both automatic and human evaluations.

With the fast growth of demand for the ubiquitous, precise, and instant indoor location information, the Received Signal Strength (RSS) based Wi-Fi indoor localization has been greeted with an avalanche of publicity. The studies in this field so far rarely consider the diversity of Wi-Fi signals, and thereby the RSS measures involving gross error on account of the complicated indoor environment deteriorate localization accuracy. In response to this compelling problem, we propose to use the concept of Asymptotic Relative Efficiency (ARE) to design a new synergetic test of Wi-Fi signal for indoor localization. Specifically, first of all, the Jarque-Bera (JB) test is conducted to test the normality of Wi-Fi signals at each Reference Point (RP). Second, the result of JB test is fed into the synergetic Mann-Whitney U and T test to construct the set of matching RPs corresponding to the newly-collected RSS data. Finally, the location coordinate of the target is obtained by calculating the K-nearest neighbor of matching RPs. Furthermore, the experimental results demonstrate that the proposed approach is featured with higher localization accuracy compared with the existing Wi-Fi indoor localization approaches.

The Internet of Things (IoT) has gradually changed the way of people’s lives due to its ability of connecting everything together, and meanwhile the accurate location sensing plays a crucial role in achieving this goal. Up to now, as one of the most representative outdoor localization systems, the global positioning system has been widely used, but its performance may be dramatically declined in indoor environment due to the serious multipath effect and signal attenuation caused by the complicated indoor structure. At the same time, the location fingerprint-based localization approach has become a popular one in indoor environment, and meanwhile the corresponding calibrated signal simplification in location database construction has been primarily considered due to its significant practical meaning in avoiding the blind signal sampling. In this paper, we propose to use an information-theoretic lens to construct the energy-efficient location fingerprint database for the localization in IoT. Interestingly, by analyzing the information loss in signal sampling, we analogize the database construction process into the information propagation process in a lossy channel, and then formulate the relations of sample capacity and localization error from an information-theoretic view. After that, by selecting an appropriate time interval to sample the independent and nonredundant signal, the minimum number of sampled signal under the given expected localization accuracy is determined. Finally, the extensive experimental results show that compared with the state-of-the-art approaches, the proposed one can effectively simplify the calibrated data for the energy-efficient location database construction in different wireless localization networks.

The rapid development of ubiquitous and high-speed wireless communication technology has driven the increasingly serious demand for the Location-based Services (LBSs). In this circumstance, we propose a new crowd-sourced calibration-free and inertial sensor- independent indoor pedestrian localization approach, namely Mann-Whitney rank-sum testing via segments labeling (Marvel). In concrete terms, first of all, the motion paths are modeled by using the A* algorithm with the floor plan provided by the merchant, and then each motion path is segmented according to the preset expected localization accuracy. Second, by setting the signal similarity threshold, the Received Signal Strength (RSS) sequences which are collected by the human subjects following their daily routines in target environment are also segmented. Third, the proposed Marvel is adopted to cluster the motion path segments as well as RSS sequence segments respectively to construct the physical and signal logic graphs. Finally, by using the concept of backbone nodes diffusion mapping to establish the mapping relations between the physical and signal spaces, the pedestrian localization and the related motion analysis are conducted by the server. Furthermore, the extensive experimental results show that the proposed approach is capable of achieving higher localization accuracy compared with the current state-of-the-art approaches.

Applications on Location Based Services (LBSs) have attracted significant attention due to its personalized, convenient, and smart user experience, and meanwhile the accurate mapping and localization algorithm plays a crucial role in satisfying the LBSs. At the same time, motivated by the widely-deployed Wi-Fi network, the Wi-Fi signal based localization has become one of the superior positioning techniques in indoor environment, and the corresponding sample capacity involved in positioning database establishment should be given much attention due to its significant guidance meaning in practice. In this paper, we propose a new sample capacity optimization approach for indoor Wi-Fi localization from the information-theoretic view, namely Sample Capacity Optimization for Positioning database Establishment (SCOPE) in indoor Wi-Fi environment. Interestingly, we analogize the positioning database establishment process in indoor Wi-Fi environment into the information propagation process in a lossy channel, and meanwhile formulate the relations between the sample capacity and localization error. Experimental result shows that the proposed SCOPE can accurately estimate the minimum sample capacity with a given expected localization accuracy under different Access Point (AP) combination.

Applications on Location Based Services (LBSs) have driven the increasing demand for indoor localization technology. The conventional location fingerprinting based localization involves heavy time and labor cost for database construction, while the well‐known Simultaneous Localization and Mapping (SLAM) technique requires assistant motion sensors as well as complicated data fusion algorithms. To solve the above problems, a new pedestrian motion learning based indoor Wireless Local Area Network (WLAN) localization approach is proposed in this paper to achieve satisfactory LBS without the demand for location calibration or motion sensors. First of all, the concept of pedestrian motion learning is adopted to construct users’ motion paths in the target environment. Second, based on the timestamp relation of the collected Received Signal Strength (RSS) sequences, the RSS segments are constructed to obtain the signal clusters with the newly defined high‐dimensional linear distance. Third, the PageRank algorithm is performed to establish the hotspot mapping relations between the physical and signal spaces which are then used to localize the target. Finally, the experimental results show that the proposed approach can effectively estimate the target’s locations and analyze users’ motion preference in indoor environment.

With the rapid development of Wireless Local Area Network (WLAN) technique, the indoor WLAN localization has caught significant attention. In this paper, a novel indoor WLAN localization approach by using the high-dimensional manifold alignment with limited calibration load is proposed. Different from the conventional dimension-reduction based manifold alignment approach which preserves a limited part of the Received Signal Strength (RSS) data information, we first construct an innovative objective function from the augmented physical locations and the corresponding RSS data. Second, the closed-form solution to the objective function is obtained by applying the Lagrange multiplier approach. Finally, the target location is estimated at the closest point in the manifold. Furthermore, we present some preliminary analysis towards the generalization of the proposed objective function to the scenario with multiple types of measurements used for the localization. The extensive analytical and experimental results demonstrate that the performance of the proposed approach is well with limited calibration load and can be further improved by using more calibrated locations with known RSS data.