Existing knowledge graphs (KGs) inevitably contain outdated or erroneous knowledge that needs to be removed from knowledge graph embedding (KGE) models. To address this challenge, knowledge unlearning can be applied to eliminate specific information while preserving the integrity of the remaining knowledge in KGs. Existing unlearning methods can generally be categorized into exact unlearning and approximate unlearning. However, exact unlearning requires high training costs, while approximate unlearning faces two issues when applied to KGs due to the inherent connectivity of triples: (1) It fails to fully remove targeted information, as forgetting triples can still be inferred from remaining ones. (2) It focuses on local data for specific removal, which weakens the remaining knowledge in the forgetting boundary. To address these issues, we propose GraphDPO, a novel approximate unlearning framework based on direct preference optimization (DPO). Firstly, to effectively remove forgetting triples, we reframe unlearning as a preference optimization problem, where the model is trained by DPO to prefer reconstructed alternatives over the original forgetting triples. This formulation penalizes reliance on forgettable knowledge, mitigating incomplete forgetting caused by KG connectivity. Moreover, we introduce an out-boundary sampling strategy to construct preference pairs with minimal semantic overlap, weakening the connection between forgetting and retained knowledge. Secondly, to preserve boundary knowledge, we introduce a boundary recall mechanism that replays and distills relevant information both within and across time steps. We construct eight unlearning datasets across four popular KGs with varying unlearning rates. Experiments show that GraphDPO outperforms state-of-the-art baselines by up to 10.1% in MRR_Avg and 14.0% in MRR_F1. Further analysis confirms that GraphDPO more effectively removes target knowledge while preserving surrounding context.

The multi-label text classification (MLTC) task involves associating text data with multiple relevant labels. However, previous studies often overlooked the co-occurrence information of labels within text, resulting in the inability to distinguish similar labels. Moreover, these studies have underestimated the importance of label node initialization. To tackle these challenges, we propose a Label-aware Multi-semantic Fusion Network (LMFN). Our approach employs label-guided attention to learn text representations closely aligned with labels. Then, in order to obtain more refined semantic representation, the word embedding matrix is introduced to integrate features from diverse sources. Concurrently, we use joint learning network to extract label features. We first initialize label nodes and use multi-layer GCNs to capture the dependencies and higher-order information between labels. Following this, cross-attention is utilized to effectively integrate label features with internal semantics and reveal latent correlations. Comprehensive experiments on two standard datasets show that our proposed model LMFN outperforms existing methods.

Fact-checking is a highly challenging task that requires verifying the truthfulness of a claim based on multiple evidence sentences. Despite the effectiveness of existing methods, they overlook the differences in the importance of various news entities. Additionally, they fail to consider the semantic relationships between the claim and the evidence from multiple perspectives. To address these issues, we propose an Entity-aware Multi-perspective Semantic Fusion Network (EMSFN) for Fact-checking Fake News Detection. First, we introduce the Entity Attention Network to extract semantic information from claim and calculate the differences in the importance of entities. Then, we built the Multi-view Semantic Relation Extraction Network to capture the interactions between claim and evidence, extracting multi-dimensional interaction information. The proposed EMSFN calculates the contribution degree of different entities and facilitates information interaction between claim and evidence from multiple perspectives. Experiments on Snopes and PolitiFact datasets validate the effectiveness of our EMSFN.

Retrieval-Augmented Generation (RAG) has emerged as a key paradigm for enhancing large language models by incorporating external knowledge.However, current RAG methods exhibit limited capabilities in complex RAG scenarios and suffer from limited task diversity.To address these limitations, we propose RAG-Instruct, a general method for synthesizing diverse and high-quality RAG instruction data based on any source corpus.Our approach leverages (1) five RAG paradigms, which encompass diverse query-document relationships, and (2) instruction simulation, which enhances instruction diversity and quality by utilizing the strengths of existing instruction datasets.Using this method, we construct a 40K instruction dataset from Wikipedia, comprehensively covering diverse RAG scenarios and tasks.Experiments demonstrate that RAG-Instruct effectively enhances LLMs' RAG capabilities, achieving strong zero-shot performance and outperforming various RAG baselines.The code is publicly available at https://github.com/FreedomIntelligence/RAG- Instruct.

RAG systems consist of multiple modules to work together. However, these modules are usually separately trained. We argue that a system like RAG that incorporates multiple modules should be jointly optimized to achieve optimal performance. To demonstrate this, we design a specific pipeline called SmartRAG that includes a policy network and a retriever. The policy network can serve as 1) a decision maker that decides when to retrieve, 2) a query rewriter to generate a query most suited to the retriever and 3) an answer generator that produces the final response with/without the observations. We then propose to jointly optimize the whole system using a reinforcement learning algorithm, with the reward designed to encourage the system to achieve the highest performance with minimal retrieval cost. When jointly optimized, each module can be aware of how other modules are working and thus find the best way to work together as a complete system. Empirical results demonstrate that the jointly optimized system can achieve better performance than separately optimized counterparts.

With the rapid development of social networks, the proliferation of fake news has become a pressing global issue. Such misleading content is often fabricated through the integration of multimodal data, including text and images, leading to detrimental effects on both society and the economy. Despite significant advancements in fake news detection methods in recent years, most existing approaches primarily emphasize global semantic features, often neglecting the critical role of local semantic features in the detection process. Furthermore, when integrating multimodal features, these methods struggle to effectively capture intermodal associations, resulting in suboptimal fusion performance. To address these challenges, we propose a novel fake news detection model (CSMDF) for cross-semantic multimodal data fusion. Initially, feature encoders are utilized to extract global semantic features from both the text and image modalities of the target news. Subsequently, we design a local semantic feature extraction module to capture the local semantic features within each modality by utilizing clustering algorithms, central loss functions, fully connected layers, and other methods. Meanwhile, a feature fusion module based on a bidirectional gated recurrent unit (BiGRU) is employed to integrate both global and local semantic features, generating cross-semantic representations for each modality. Finally, we implement a Co-Attention mechanism to facilitate multimodal feature fusion by integrating cross-semantic features, establishing intermodal associations, and capturing their interactive relationships. Experimental results on real datasets demonstrate that CSMDF consistently outperforms state-of-the-art methods, enhancing multimodal fake news detection.

Improving the reasoning capabilities of large language models (LLMs) typically requires supervised fine-tuning with labeled data or computationally expensive sampling. We introduce Unsupervised Prefix Fine-Tuning (UPFT), which leverages the observation of Prefix Self-Consistency -- the shared initial reasoning steps across diverse solution trajectories -- to enhance LLM reasoning efficiency. By training exclusively on the initial prefix substrings (as few as 8 tokens), UPFT removes the need for labeled data or exhaustive sampling. Experiments on reasoning benchmarks show that UPFT matches the performance of supervised methods such as Rejection Sampling Fine-Tuning, while reducing training time by 75\% and sampling cost by 99\%. Further analysis reveals that errors tend to appear in later stages of the reasoning process and that prefix-based training preserves the model’s structural knowledge. This work demonstrates how minimal unsupervised fine-tuning can unlock substantial reasoning gains in LLMs, offering a scalable and resource-efficient alternative to conventional approaches.

Recent advancements in Long Chain-of-Thought (CoT) reasoning models have improved performance on complex tasks, but they suffer from overthinking, which generates redundant reasoning steps, especially for simple questions. This paper revisits the reasoning patterns of Long and Short CoT models, observing that the Short CoT patterns offer concise reasoning efficiently, while the Long CoT patterns excel in challenging scenarios where the Short CoT patterns struggle. To enable models to leverage both patterns, we propose Question-Free Fine-Tuning (QFFT), a fine-tuning approach that removes the input question during training and learns exclusively from Long CoT responses. This approach enables the model to adaptively employ both reasoning patterns: it prioritizes the Short CoT patterns and activates the Long CoT patterns only when necessary. Experiments on various mathematical datasets demonstrate that QFFT reduces average response length by more than 50\%, while achieving performance comparable to Supervised Fine-Tuning (SFT). Additionally, QFFT exhibits superior performance compared to SFT in noisy, out-of-domain, and low-resource scenarios.

With the success of pre-trained visual-language (VL) models such as CLIP in visual representation tasks, transferring pre-trained models to downstream tasks has become a crucial paradigm. Recently, the prompt tuning paradigm, which draws inspiration from natural language processing (NLP), has made significant progress in VL field. However, preceding methods mainly focus on constructing prompt templates for text and visual inputs, neglecting the gap in class label representations between the VL models and downstream tasks. To address this challenge, we introduce an innovative label alignment method named \textbf{LAMM}, which can dynamically adjust the category embeddings of downstream datasets through end-to-end training. Moreover, to achieve a more appropriate label distribution, we propose a hierarchical loss, encompassing the alignment of the parameter space, feature space, and logits space. We conduct experiments on 11 downstream vision datasets and demonstrate that our method significantly improves the performance of existing multi-modal prompt learning models in few-shot scenarios, exhibiting an average accuracy improvement of 2.31(\%) compared to the state-of-the-art methods on 16 shots. Moreover, our methodology exhibits the preeminence in continual learning compared to other prompt tuning methods. Importantly, our method is synergistic with existing prompt tuning methods and can boost the performance on top of them. Our code and dataset will be publicly available at https://github.com/gaojingsheng/LAMM.

Traditional knowledge graph embedding (KGE) methods typically require preserving the entire knowledge graph (KG) with significant training costs when new knowledge emerges. To address this issue, the continual knowledge graph embedding (CKGE) task has been proposed to train the KGE model by learning emerging knowledge efficiently while simultaneously preserving decent old knowledge. However, the explicit graph structure in KGs, which is critical for the above goal, has been heavily ignored by existing CKGE methods. On the one hand, existing methods usually learn new triples in a random order, destroying the inner structure of new KGs. On the other hand, old triples are preserved with equal priority, failing to alleviate catastrophic forgetting effectively. In this paper, we propose a competitive method for CKGE based on incremental distillation (IncDE), which considers the full use of the explicit graph structure in KGs. First, to optimize the learning order, we introduce a hierarchical strategy, ranking new triples for layer-by-layer learning. By employing the inter- and intra-hierarchical orders together, new triples are grouped into layers based on the graph structure features. Secondly, to preserve the old knowledge effectively, we devise a novel incremental distillation mechanism, which facilitates the seamless transfer of entity representations from the previous layer to the next one, promoting old knowledge preservation. Finally, we adopt a two-stage training paradigm to avoid the over-corruption of old knowledge influenced by under-trained new knowledge. Experimental results demonstrate the superiority of IncDE over state-of-the-art baselines. Notably, the incremental distillation mechanism contributes to improvements of 0.2%-6.5% in the mean reciprocal rank (MRR) score. More exploratory experiments validate the effectiveness of IncDE in proficiently learning new knowledge while preserving old knowledge across all time steps.

Large language models (LLMs) have demonstrated impressive proficiency in information retrieval, while they are prone to generating incorrect responses that conflict with reality, a phenomenon known as intrinsic hallucination. The critical challenge lies in the unclear and unreliable fact distribution within LLMs trained on vast amounts of data. The prevalent approach frames the factual detection task as a question-answering paradigm, where the LLMs are asked about factual knowledge and examined for correctness. However, existing studies primarily focused on deriving test cases only from several specific domains, such as movies and sports, limiting the comprehensive observation of missing knowledge and the analysis of unexpected hallucinations. To address this issue, we propose OntoFact, an adaptive framework for detecting unknown facts of LLMs, devoted to mining the ontology-level skeleton of the missing knowledge. Specifically, we argue that LLMs could expose the ontology-based similarity among missing facts and introduce five representative knowledge graphs (KGs) as benchmarks. We further devise a sophisticated ontology-driven reinforcement learning (ORL) mechanism to produce error-prone test cases with specific entities and relations automatically. The ORL mechanism rewards the KGs for navigating toward a feasible direction for unveiling factual errors. Moreover, empirical efforts demonstrate that dominant LLMs are biased towards answering Yes rather than No, regardless of whether this knowledge is included. To mitigate the overconfidence of LLMs, we leverage a hallucination-free detection (HFD) strategy to tackle unfair comparisons between baselines, thereby boosting the result robustness. Experimental results on 5 datasets, using 32 representative LLMs, reveal a general lack of fact in current LLMs. Notably, ChatGPT exhibits fact error rates of 51.6% on DBpedia and 64.7% on YAGO, respectively. Additionally, the ORL mechanism demonstrates promising error prediction scores, with F1 scores ranging from 70% to 90% across most LLMs. Compared to the exhaustive testing, ORL achieves an average recall of 80% while reducing evaluation time by 35.29% to 63.12%.

The in-context learning (ICL) for relational triple extraction (RTE) has achieved promising performance, but still encounters two key challenges: (1) how to design effective prompts and (2) how to select proper demonstrations. Existing methods, however, fail to address these challenges appropriately. On the one hand, they usually recast RTE task to text-to-text prompting formats, which is unnatural and results in a mismatch between the output format at the pre-training time and the inference time for large language models (LLMs). On the other hand, they only utilize surface natural language features and lack consideration of triple semantics in sample selection. These issues are blocking improved performance in ICL for RTE, thus we aim to tackle prompt designing and sample selection challenges simultaneously. To this end, we devise a tabular prompting for RTE (TableIE) which frames RTE task into a table generation task to incorporate explicit structured information into ICL, facilitating conversion of outputs to RTE structures. Then we propose instructive in-context learning (I^2CL) which only selects and annotates a few samples considering internal triple semantics in massive unlabeled samples. Specifically, we first adopt off-the-shelf LLMs to perform schema-agnostic pre-extraction of triples in unlabeled samples using TableIE. Then we propose a novel triple-level similarity metric considering triple semantics between these samples and train a sample retrieval model based on calculated similarities in pre-extracted unlabeled data. We also devise three different sample annotation strategies for various scenarios. Finally, the annotated samples are considered as few-shot demonstrations in ICL for RTE. Experimental results on two RTE benchmarks show that I^2CL with TableIE achieves state-of-the-art performance compared to other methods under various few-shot RTE settings.

Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Zhenyang Cai, Ke Ji, Xiang Wan, Benyou Wang. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

Social media platforms have emerged as the primary channels for the general public to access and share information. However, the rapid dissemination of news has led to the emergence of a significant amount of unverified content, posing a serious threat to media credibility and network security. The current solutions primarily focus on detecting fake news by extracting and fusing features from images and text, but they have not fully utilized the relevance within and between modalities, resulting in suboptimal fusion effects. In this paper, we propose a multimodal fusion framework via multi-attention mechanism (MFMA), which considers not only the semantic and relevance features of images, textual content, and Optical Character Recognition (OCR) text from the news, but also the multimodal relevance features between them. First, we extract text within images (OCR text) to address the issue of inadequate utilization of the visual modality in traditional methods. Second, we select more robust feature extractors to capture the independent characteristics of each modality and use a pre-trained FcaNet model to extract more comprehensive image quality features. Additionally, two relevance extraction modules are designed to achieve feature fusion within and between modalities. Finally, the combined features are fed into a classifier to determine the authenticity of the news. The experimental results and analysis indicate that the model we proposed effectively enhances the performance of fake news detection.

Low-resource relation extraction (LRE) aims to extract relations from limited labeled corpora. Existing work takes advantages of self-training or distant supervision to expand the limited labeled data in the data-driven approaches, while the selection bias of pseudo labels may cause the error accumulation in subsequent relation classification. To address this issue, this paper proposes fmLRE, an iterative feedback method based on feature mapping similarity calculation to improve the accuracy of pseudo labels. First, it calculates the similarities between pseudo-label and real-label data of the same category in a feature mapping space based on semantic features of labeled dataset after feature projection. Then, it fine-tunes initial model according to the iterative process of reinforcement learning. Finally, the similarity is used as a threshold for screening high-precision pseudo-labels and the basis for setting different rewards, which also acts as a penalty term for the loss function of relation classifier. Experimental results demonstrate that fmLRE achieves the state-of-the-art performance compared with strong baselines on two public datasets.

Due to the complex label hierarchy and intensive labeling cost in practice, the hierarchical text classification (HTC) suffers a poor performance especially when low-resource or fewshot settings are considered.Recently, there is a growing trend of applying prompts on pretrained language models (PLMs), which has exhibited effectiveness in the few-shot flat text classification tasks.However, limited work has studied the paradigm of prompt-based learning in the HTC problem when the training data is extremely scarce.In this work, we define a path-based few-shot setting and establish a strict path-based evaluation metric to further explore few-shot HTC tasks.To address the issue, we propose the hierarchical verbalizer ("Hi-erVerb"), a multi-verbalizer framework treating HTC as a single-or multi-label classification problem at multiple layers and learning vectors as verbalizers constrained by hierarchical structure and hierarchical contrastive learning.In this manner, HierVerb fuses label hierarchy knowledge into verbalizers and remarkably outperforms those who inject hierarchy through graph encoders, maximizing the benefits of PLMs.Extensive experiments on three popular HTC datasets under the few-shot settings demonstrate that prompt with HierVerb significantly boosts the HTC performance, meanwhile indicating an elegant way to bridge the gap between the large pre-trained model and downstream hierarchical classification tasks. 1

Abstract In the era of the explosion of digital content of large‐scale self‐media, user‐friendly social platforms such as Twitter and Facebook, provide opportunities for people to express their ideas and opinions freely. Due to lack of restrictions, hateful speech and its exposure can have profound psychological impacts on society. Current social networking platform is over‐reliant on the manual check, and it is labor‐intensive and time‐consuming. Although there are many machines learning methods for the detection of hate speech, short text with character limit on social platforms is more challenging for the detection of hate speech and offensive language. To address the problem of data sparsity, we have proposed a topic memory model for hate speech and offensive language detection (abbreviated as TM‐HOL). Potential topics are generated with our encoder and decoder to enrich short text features. Two memory matrices correspond to the topic words and the text, and the hate feature matrix is used to learn the syntactic features. It is demonstrated that our proposed method is effective on three datasets, performing better weighted‐F1.

Distributed clustering based on the Gaussian mixture model (GMM) has exhibited excellent clustering capabilities in peer-to-peer (P2P) networks. However, more iterative numbers and communication overhead are required to achieve the consensus in existing distributed GMM clustering algorithms. In addition, the truth that it cannot find a closed form for the update of parameters in GMM causes the imprecise clustering accuracy. To solve these issues, by utilizing the transfer learning technique, a general transfer distributed GMM clustering framework is exploited to promote the clustering performance and accelerate the clustering convergence. In this work, each node is treated as both the source domain and the target domain, and these nodes can learn from each other to complete the clustering task in distributed P2P networks. Based on this framework, the transfer distributed expectation-maximization algorithm with the fixed learning rate is first presented for data clustering. Then, an improved version is designed to obtain the stable clustering accuracy, in which an adaptive transfer learning strategy is adopted to adjust the learning rate automatically instead of a fixed value. To demonstrate the extensibility of the proposed framework, a representative GMM clustering method, the entropy-type classification maximum-likelihood algorithm, is further extended to the transfer distributed counterpart. Experimental results verify the effectiveness of the presented algorithms in contrast with the existing GMM clustering approaches.

Collaborative multiview clustering methods can efficiently realize the view fusion by exploring complementary and consistent information among multiple views. However, these studies ignore all the differences between multiple views in fusion. In fact, in the multiview clustering, the data are diverse from view to view. The larger the difference between any two views is, the more the fusion of these views is required. Moreover, a global tradeoff parameter is generally adopted to restrain the penalty related to the disagreement of all views, which is often defined empirically. Inspired by the idea of transfer learning, a series of novel collaborative multiview clustering algorithms are proposed to tackle these challenges. In the most basic one, each view performs clustering independently and learns from others to improve its own clustering performance, in which a global learning factor is defined to control the interaction between multiple views. The fuzzy memberships are regarded as the important knowledge to provide guidance between views, and the consensus constraint is defined to ensure the consistent partitions of all views. In addition, the local adaptive learning factors between any two views instead of a global fixed one are adopted in an improved version to emphasize the difference between views, and the adjustment strategy for the learning factor is further designed to guarantee the stability of multiview clustering without the influence of initial values. Finally, to identify the significance of different views to the clustering, the extended versions are excavated with the assignment of view weights and the maximum entropy regularization technique is employed to optimize the weights. Experiments on various real-world multiview datasets verify the superiority of the presented approaches.

In recent years, kernel-based fuzzy clustering has attracted significant attention, primarily benefiting from the outstanding performance of capturing the potential non-linear structure in data clustering. However, many existing kernel clustering methods are not available for large datasets due to computational costs. To overcome this limitation, the low-rank random feature map is utilized to approximate the kernel space. Nevertheless, this kind of feature approximation method ignores the graph structure information hidden in the data and does not take the correlations between data samples in the clustering into account. Thus, we present a new kernel fuzzy clustering based on Quasi-Monte Carlo feature map with neighbor affinity constraint (Na_QMC_KFC). In this scheme, the Quasi-Monte Carlo method is adopted to approximate the Gaussian kernel function so as to reduce the computational costs. Meanwhile, the neighbor affinity constraint is designed to maintain the graph structure information of the data and further facilitate the consistency of the membership degrees and the raw data. What’s more, the Alternating Direction Method of Multipliers method is utilized to optimize the problem with respect to the neighbor affinity lasso. The experiments on several non-linear and real-world datasets exhibits the efficiency of the presented algorithm.

The repeaters in cascade structure can be used to suppress the cross coupling between the nonadjacent repeaters in multiload inductive power transfer (ML-IPT) system. In this article, the cascade repeaters used in the ML-IPT system work both as power transmitters and load carriers. Based on the cascade scheme, the system models of two ML-IPT systems with single-excitation (SE) scheme for different load position are established and high-order compensations are designed to achieve constant load currents. Referring to the topologies of the two ML-IPT systems with SE scheme, a bilateral-excitation (BE) scheme is further proposed to improve output currents and system reliability, and the comparison of the conventional scheme and the proposed scheme is also presented. Finally, a prototype of ML-IPT system with the proposed BE scheme is built, where the maximum deviation of the output currents is less than 13% and the maximum power transfer efficiency reaches 89.2% for an eight-load case.

This paper presents an optimization strategy for the passive thermosiphon beam system to maintain the indoor thermal comfort and reduce the energy consumption. The terminal unit model, indoor built model and energy models of the components in the passive thermosiphon beam (PTB) system are developed respectively. To obtain the optimal operating points, the optimization problem is formulated with respect to the system constrains and components interactions. The genetic algorithm is utilized to find the optimal fresh air supply and chilled water flow rate of the PTB system. The result indicates that the optimization strategy can reduce the energy consumption of the PTB system by 9.3%.

In recent years, the scale and diversity of malicious software on mobile networks are constantly increasing, thereby causing considerable danger to users' property and personal privacy. In this study, we devise a method that uses the URLs visited by applications to identify malicious apps. A multi-view neural network is used to create a malware detection model that emphasizes depth and width. This neural network can create multiple views of the input automatically and distribute soft attention weights to focus on different features of input. Multiple views preserve rich semantic information from input for classification without requiring complicated feature engineering. In addition, we conduct comprehensive experiments to compare the proposed method with others and verify the validity of the detection model. The experimental results show that our method has a certain timeliness. It can not only effectively detect malware discovered in different months of a certain year, but also detect potentially malicious apps in the third-party app market. We also compare the detection results of the proposed method on wild apps with 10 popular anti-virus scanners, and the final result shows that our approach ranks second in terms of detection performance.