Spatial stratified heterogeneity refers to the pattern variation of the target phenomenon across different regions. Current measures mainly quantify spatially stratified heterogeneity in terms of the consistency between strata and the target variable and neglect the complexity of stratification, whereas complex stratification may lead to overfitting and an overestimated degree of heterogeneity. To address this issue, this paper enhances the relative-entropy-based spatial stratified heterogeneity measure to unit explanatory power using the entropy of the stratification. The proposed method first quantifies the stratification complexity using the minimum theoretical number of bits for encoding it, then characterizes the degree of heterogeneity per bit, i.e., unit explanatory power, by the quotient of the relative-entropy-based measure and the stratification complexity. Additionally, this paper develops two visualization tools for interpreting and comparing unit explanatory power and reveals the relation among spatial stratified heterogeneity, stratification complexity, and the log-likelihood function. Finally, we conduct experiments on both illustrative and real-life data sets to show the advantages of the unit explanatory power over traditional spatial stratified heterogeneity measures. The code for computing unit explanatory power has been released athttps://github.com/crafly/ssh_with_gran.

The performance of multimodal models often deteriorates when modality absence occurs. The absence disrupts the learned inter-modal correlations, resulting in biased multimodal representations. This challenge is especially pronounced when the absence is pervasive, affecting both the training and inference phases. Recent studies have attempted to reconstruct the missing information; however, most of them require complete supervision, which is seldom available in scenarios of pervasive absence. The quality of reconstruction remains a critical issue. Alternatively, others aim to learn robust representations from the available modalities but the substantial variations and biases are not fully addressed. This paper introduces the Multimodal Generalization and Refinement (MGR) framework to mitigate the issue of pervasive modality absence. MGR begins by acquiring generalized multimodal representations and iteratively refines them to recognize and calibrate the biased representations. Initially, multimodal samples with absence are embedded through foundation models, and MGR integrates independent unimodal features to further enhance generalization. Additionally, a novel mixed-context prompt is adopted to identify biases in both features and correlations. A redistribution operation can then refine these biases through graph pooling, culminating in robust and calibrated multimodal representations, which are suitable for downstream tasks. Comprehensive experiments on four benchmark datasets demonstrate that the proposed MGR framework outperforms state-of-the-art methods, effectively mitigating the impact of pervasive modality absence.

Joint relation extraction models effectively mitigate the error propagation problem inherently present in pipeline models. Nevertheless, joint models face challenges including high computational complexity, complex network architectures, difficult parameter tuning, and notably, limited interpretability. In contrast, recent advances in pipeline relation extraction models (PURE, PL-Marker) have attracted considerable attention due to their lightweight design and high extraction accuracy. A key advancement is the introduction of a marker mechanism, which enhances relation extraction (RE) process by highlighting entities. However, these models primarily focus on generating correct labels. In doing so, they neglect the label selection process. Moreover, they fail to adequately capture the intricate interactions between entity pairs. To overcome these limitations, we develop a Candidate Label Markers (CLMs) mechanism that prioritizes strategic label selection over simple label generation. Furthermore, we facilitate interactions among diverse relation pairs, enabling the identification of more intricate relational patterns. Experimental results show that we achieve a new SOTA performance. Specifically, based on the same Named Entity Recognition (NER) results as theirs, we improve the SOTA methods by 2.5%, 1.9%, 1.2% in terms of strict F1 scores on SciERC, ACE05 and ACE04.

Federated Hetero-Task Learning (FHTL) enables the simultaneous learning of multiple heterogeneous tasks on federated learning clients, offering enhanced flexibility. However, the inconsistency between optimization objectives and evaluation metrics for these heterogeneous tasks poses challenges in achieving performance fairness among clients. This study proposes a fairness-aware FHTL method, FairFHTL. It employs adversarial multi-task representation learning at the client level to learn the task-independent shared model. Consequently, it solves optimization objectives inspired by fair resource allocation on the server side to determine the update direction of the global shared model, ultimately achieving task-independent fair performance balance. Extensive experiments on three multi-task datasets demonstrate that FairFHTL significantly enhances performance across the majority of tasks compared to conventional federated learning and FHTL methods. Moreover, compared with other fairness-aware federated learning approaches, FairFHTL maintains a more uniform performance distribution across all tasks.

Clinical event extraction involves extracting event attributes from clinical medical records. However, arguments involved in clinical events exhibit specificity, diversity and ambiguity, posing substantial challenges for existing models. The scarcity of Chinese clinical datasets further impedes research on clinical event extraction. Existing models commonly experience issues of contextual information decay during multi-task processes. Furthermore, unlike entities in general domains, medical entities are expressed in complex ways, resulting in low recall. To address these challenges, we propose a Clinical Event Extraction model based on Contextual ResiduAl and Soft Encoding Relation Extraction(CARE), which consists of an encoding module, a relation detection module, and an entity recognition module. The relation detection module identifies potential relations within a medical record. To prevent error propagation in relation extraction, a soft encoding strategy is proposed to discern target relations from candidate ones. The entity recognition module employs the contextual residual connection mechanism to concatenate the text with relation between semantic templates before feeding them into the entity recognition module. On CHIP-CDEE and CEMRs, CARE achieves F1 scores of 73.22% and 94.83%, respectively, which outperforms all baseline models, including LLMs, demonstrating its effectiveness for this task.

Large Language Models (LLMs) demonstrate remarkable potential in Event Argument Extraction (EAE) tasks due to their powerful capabilities in contextual understanding and semantic generation. However, the absence of event schema knowledge limits their performance in these tasks. Additionally, we observe a strong correlation between argument roles and entity types, which is often disregarded in prevailing models. To address these challenges, we propose FINE, a prompt tuning approach Fused with InterNal and External knowledge for low-resource EAE based on generative framework with LLM, which integrates global event schema knowledge and local entity information. Specifically, FINE employs External Knowledge (EK) Prompt Generator to construct external knowledge prompts with high event-awareness, which can help better capture the semantics of roles and their diversity among various events. Furthermore, we propose RAEA, a Role-Associated Entity Argument candidate mechanism to filter relevant entities within the context, effectively reducing interference from irrelevant entities. Experimental results on ACE05-EN and ERE-EN datasets demonstrate that our proposed FINE model achieves significant improvements in EAE, particularly in low-resource scenarios.

Large Language Models (LLMs) have showcased exceptional capabilities in various domains, attracting significant interest from both academia and industry. Despite their impressive performance, the substantial size and computational demands of LLMs pose considerable challenges for practical deployment, particularly in environments with limited resources. The endeavor to compress language models while maintaining their accuracy has become a focal point of research. Among the various methods, knowledge distillation has emerged as an effective technique to enhance inference speed without greatly compromising performance. This article presents a thorough survey from three aspects: method, evaluation, and application, exploring knowledge distillation techniques tailored specifically for LLMs. Specifically, we divide the methods into white-box KD and black-box KD to better illustrate their differences. Furthermore, we also explored the evaluation tasks and distillation effects between different distillation methods and proposed directions for future research. Through in-depth understanding of the latest advancements and practical applications, this survey provides valuable resources for researchers, paving the way for sustained progress in this field.

Federated Foundation Models (FedFMs) represent a distributed learning paradigm that fuses general competences of foundation models as well as privacy-preserving capabilities of federated learning. This combination allows the large foundation models and the small local domain models at the remote clients to learn from each other in a teacher-student learning setting. This paper provides a comprehensive summary of the ten challenging problems inherent in FedFMs, encompassing foundational theory, utilization of private data, continual learning, unlearning, Non-IID and graph data, bidirectional knowledge transfer, incentive mechanism design, game mechanism design, model watermarking, and efficiency. The ten challenging problems manifest in five pivotal aspects: “Foundational Theory,” which aims to establish a coherent and unifying theoretical framework for FedFMs. “Data,” addressing the difficulties in leveraging domain-specific knowledge from private data while maintaining privacy; “Heterogeneity,” examining variations in data, model, and computational resources across clients; “Security and Privacy,” focusing on defenses against malicious attacks and model theft; and “Efficiency,” highlighting the need for improvements in training, communication, and parameter efficiency. For each problem, we offer a clear mathematical definition on the objective function, analyze existing methods, and discuss the key challenges and potential solutions. This in-depth exploration aims to advance the theoretical foundations of FedFMs, guide practical implementations, and inspire future research to overcome these obstacles, thereby enabling the robust, efficient, and privacy-preserving FedFMs in various real-world applications.

Federated learning (FL) enables collaborative learning across multiple biomedical data silos with multimodal foundation models while preserving privacy. Due to the heterogeneity in data processing and collection methodologies across diverse medical institutions and the varying medical inspections patients undergo, modal heterogeneity exists in practical scenarios, where severe modal heterogeneity may even prevent model training. With privacy considerations, data transfer cannot be permitted, restricting knowledge exchange among different clients. To trickle these issues, we propose a cross-modal prototype imputation method for visual-language understanding (Buffalo) with only a slight increase in communication cost, which can improve the performance of fine-tuning general foundation models for downstream biomedical tasks. We conducted extensive experiments on medical report generation and biomedical visual question-answering tasks. The results demonstrate that Buffalo can fully utilize data from all clients to improve model generalization compared to other modal imputation methods in three modal heterogeneity scenarios, approaching or even surpassing the performance in the ideal scenario without missing modality.

Federated Learning has emerged as a revolutionary innovation in the evolving landscape of global healthcare, fostering collaboration among institutions and facilitating collaborative data analysis. As practical applications continue to proliferate, numerous federations have formed in different regions. The optimization and sustainable development of federation-pretrained models have emerged as new challenges. These challenges primarily encompass privacy, population shift and data dependency, which may lead to severe consequences such as the leakage of sensitive information within models and training samples, unfair model performance and resource burdens. To tackle these issues, we propose FairFusion, a cross-federation model fusion approach that enhances privacy and fairness. FairFusion operates across federations within a Model Trip paradigm, integrating knowledge from diverse federations to continually enhance model performance. Through federated model fusion, multi-objective quantification and optimization, FairFusion obtains trustworthy solutions that excel in utility, privacy and fairness. We conduct comprehensive experiments on three public real-world healthcare datasets. The results demonstrate that FairFusion achieves outstanding model fusion performance in terms of utility and fairness across various model structures and subgroups with sensitive attributes while guaranteeing model privacy.

Multi-modality physiological signal-based emotion recognition has attracted increasing attention as its capacity to capture human affective states comprehensively. Due to multi-modality heterogeneity and cross-subject divergence, practical applications struggle with generalizing models across individuals. Effectively addressing both issues requires mitigating the gap between multimodal signals while acquiring generalizable representations across subjects. However, existing approaches often handle these dual challenges separately, resulting in suboptimal generalization. This study introduces a novel framework, termed Correlation-Driven Multi-Modality Graph Decomposition (CMMGD). The proposed CMMGD initially captures adaptive cross-modal correlations. It connects each unimodal graph to a multimodal mixed graph. To simultaneously address the dual challenges, it incorporates a correlation-driven graph decomposition module that decomposes the mixed graph into concordant and discrepant subgraphs based on the correlations. The decomposed concordant subgraph encompasses consistently activated features across modalities and subjects during emotion elicitation, unveiling a generalizable subspace. Additionally, we design a Multi-Modality Graph Regularized Transformer (MGRT) backbone specifically tailored for multimodal physiological signals. The MGRT can alleviate the over-smoothing issue and mitigate over-reliance on any single modality. Extensive experiments demonstrate that CMMGD outperforms the state-of-the-art methods by 1.79% and 2.65% on DEAP and MAHNOB-HCI datasets, respectively, under the leave-one-subject-out cross-validation strategy.

Model fusion is becoming a crucial component in the context of model-as-a-service scenarios, enabling the delivery of high-quality model services to local users. However, this approach introduces privacy risks and imposes certain limitations on its applications. Ensuring secure model exchange and knowledge fusion among users becomes a significant challenge in this setting. To tackle this issue, we propose PrivFusion, a novel architecture that preserves privacy while facilitating model fusion under the constraints of local differential privacy. PrivFusion leverages a graph-based structure, enabling the fusion of models from multiple parties without additional training. By employing randomized mechanisms, PrivFusion ensures privacy guarantees throughout the fusion process. To enhance model privacy, our approach incorporates a hybrid local differentially private mechanism and decentralized federated graph matching, effectively protecting both activation values and weights. Additionally, we introduce a perturbation filter adapter to alleviate the impact of randomized noise, thereby recovering the utility of the fused model. Through extensive experiments conducted on diverse image datasets and real-world healthcare applications, we provide empirical evidence showcasing the effectiveness of PrivFusion in maintaining model performance while preserving privacy. Our contributions offer valuable insights and practical solutions for secure and collaborative data analysis within the domain of privacy-preserving model fusion.

Physiological signal based ubiquitous computing has garnered significant attention. However, the heterogeneity among multimodal physiological signals poses a critical challenge to practical applications. To traverse this heterogeneity gap, recent studies have focused on establishing inter-modality correlations. Early works only consider coarse-level correlations between the embeddings of each modality. More recent graph-based approaches incorporate prior knowledge-based correlations, although they may not be entirely accurate. In this article, we propose the Graph Joint Fusion (GJFusion) network, which leverages channel-level inter-modality correlations based on a graph joint to mitigate the heterogeneous gap. Our proposed GJFusion first represents each modality as a graph, with each vertex corresponding to a signal channel, and the edges denoting their functional connectivity. We then join each modality by constructing inter-modality correlations for each salient channel using a sampling-based matching method. Discarded channels are transformed into a virtual vertex through a lightweight pooling operation. Subsequently, the fusion network integrates intra- and inter-modality features, enabling multimodal physiological signal fusion. To validate the effectiveness of our method, we select emotional state recognition as the downstream task and conduct comprehensive experiments on two benchmark datasets. The results demonstrate that our proposed GJFusion network surpasses the latest state-of-the-art methods, achieving relative accuracy improvements of 1.22% and 0.81% on the DEAP and MAHNOB-HCI datasets, respectively. Furthermore, visualization experiments of the salient brain regions reveal the presence of interpretable knowledge within the proposed GJFusion model.

Federated learning has emerged as a powerful technique for training robust models while preserving data privacy and security. However, real-world applications, especially in domains like healthcare, often face challenges due to non-independent and non-identically distributed (non-iid) data across different institutions. Additionally, the heterogeneity of data and the absence of a trusted central server further hinder collaborative efforts among medical institutions. Our paper introduces a novel federated learning approach called FedTAM, which incorporates cyclic model transfer and feature attention-based multi-teacher knowledge distillation. FedTAM is designed to tailor personalized models for individual clients within a decentralized federated learning setting, where data distribution is non-iid. Notably, this method enables student clients to selectively acquire the most pertinent and valuable knowledge from teacher clients through feature attention mechanism while filtering out irrelevant information. We conduct extensive experiments across five benchmark healthcare datasets and one public image classification dataset with feature shifts. Our results conclusively demonstrate that our method achieves remarkable accuracy improvements when compared to state-of-the-art approaches. This affirms the potential of FedTAM to significantly enhance federated learning performance, especially in challenging real-world contexts like healthcare.

Document-level event extraction has become one of the important research directions in natural language processing, which aims to extract the complete event arguments from the whole document. However, existing models cannot fully utilize contextual semantic relations and sequential information in complex document-level scenarios, failing to mine the relational facts between multiple sentences and arguments. To address the above problems, we propose a heterogeneous graph interaction event extraction model fusing positional embedding and attention matrix called GPAIT. Firstly, contextual semantic relationships are enhanced between entities by constructing entity attention relationship matrix; Secondly, the attention matrix is combined to rectify the representation of heterogeneous graphs; Finally, positional embedding is fused into graphs to build a graph convolutional neural network that can better capture semantic sequential relationships. Experiments on ChFinAnn and COVID-19 News show that GPAIT outperforms other document-level event extraction methods in terms of F1 by 1.2% and 3.6% respectively.

Question Answering based on Tabular and Textual data is a novel task proposed in recent years in the field of QA. At present, most QA systems return answers from a single data form, such as knowledge graphs, tables, texts. However, hybrid data including structured and unstructured data is quite pervasive in real life instead of a single form. Recent research on TAT-QA mainly suffers from the higher error of extracting supporting evidences from both tabular and textual content. This paper aimed to address the problem of failure evidence extraction from more complex and realistic hybrid data. We first proposed two types of metrics to evaluate the performance of evidence extraction on hybrid data, i.e. wrong evidence ratio (WER) and missing evidence ratio (MER). Then we utilize a candidate extractor to obtain supporting evidence related to the question. Third, an origin selector is designed to determine from where the question’s answer comes. Finally, the loss of origin selector is fused to the final loss function, which can improve the evidence extraction performance. Experimental results on the TAT-QA dataset showed that our proposed model outperforms the best baseline in terms of F1, WER and MER, which proves the effectiveness of our model.

This paper proposes a new classification method for spatial data by adjusting prior class probabilities according to local spatial patterns. First, the proposed method uses a classical statistical classifier to model training data. Second, the prior class probabilities are estimated according to the local spatial pattern and the classifier for each unseen object is adapted using the estimated prior probability. Finally, each unseen object is classified using its adapted classifier. Because the new method can be coupled with both generative and discriminant statistical classifiers, it performs generally more accurately than other methods for a variety of different spatial datasets. Experimental results show that this method has a lower prediction error than statistical classifiers that take no spatial information into account. Moreover, in the experiments, the new method also outperforms spatial auto-logistic regression and Markov random field-based methods when an appropriate estimate of local prior class distribution is used.