Xiao Liang, Zhong-Zhi Li, Zhenghao Lin, Eric Hanchen Jiang, Hengyuan Zhang, Yelong Shen, Kai-Wei Chang, Ying Nian Wu, Yeyun Gong, Weizhu Chen. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

The financial industry faces a substantial workload in verifying document images.Existing methods based on visual features struggle to identify fraudulent document images due to the lack of visual clues on the tampering region.This paper proposes CSIAD (Cross-Sample Image Anomaly Detection) by leveraging LLMs to identify logical inconsistencies in similar images.This novel framework accurately detects forged images with slight tampering traces and explains anomaly detection results.Furthermore, we introduce CrossCred, a new benchmark of real-world fraudulent images with fine-grained manual annotations.Experiments demonstrate that CSIAD outperforms state-of-the-art image fraud detection methods by 79.6% (F 1) on CrossCred and deployed industrial solutions by 21.7% (F 1) on business data.The benchmark is available at https://github.com/XMUDM/CSIAD.

The development of Large Language Models (LLMs) has revolutionized QA across various industries, including the database domain. However, there lacks a thorough evaluation regarding the capabilities of different LLMs in database QA. To this end, we introduce DQABench, the first comprehensive database QA benchmark for LLMs. DQABench features an innovative LLM-based method to automate the generation, cleaning, and rewriting of evaluation dataset, resulting in over 200,000 QA pairs in English and Chinese. These QA pairs cover a wide range of database-specific knowledge extracted from manuals, online communities, and DB instances, allowing for assessment of LLMs' Retrieval-Augmented Generation (RAG) and Tool Invocation Generation (TIG) capabilities in the database QA task. Furthermore, we propose a highly modular and scalable testbed DQATestbed, with basic and advanced components such as Fine-tuning, Question Classification Routing (QCR), RAG, TIG, and Prompt Template Engineering (PTE). Finally, we provide an evaluation pipeline that computes various metrics throughout a standardized evaluation process to ensure the accuracy and fairness. Our evaluation reveals the strengths and limitations of nine open-source and commercial LLMs, and the impact of various service components (e.g., fine-tuning, QCR, RAG, TIG). The proposed benchmark dataset is available at https://github.com/XMUDM/DQABench.

Yi Luo, Zhenghao Lin, YuHao Zhang, Jiashuo Sun, Chen Lin, Chengjin Xu, Xiangdong Su, Yelong Shen, Jian Guo, Yeyun Gong. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

Previous language model pre-training methods have uniformly applied a next-token prediction loss to all training tokens. Challenging this norm, we posit that ''Not all tokens in a corpus are equally important for language model training''. Our initial analysis examines token-level training dynamics of language model, revealing distinct loss patterns for different tokens. Leveraging these insights, we introduce a new language model called Rho-1. Unlike traditional LMs that learn to predict every next token in a corpus, Rho-1 employs Selective Language Modeling (SLM), which selectively trains on useful tokens that aligned with the desired distribution. This approach involves scoring training tokens using a reference model, and then training the language model with a focused loss on tokens with higher scores. When continual continual pretraining on 15B OpenWebMath corpus, Rho-1 yields an absolute improvement in few-shot accuracy of up to 30% in 9 math tasks. After fine-tuning, Rho-1-1B and 7B achieved state-of-the-art results of 40.6% and 51.8% on MATH dataset, respectively - matching DeepSeekMath with only 3% of the pretraining tokens. Furthermore, when continual pretraining on 80B general tokens, Rho-1 achieves 6.8% average enhancement across 15 diverse tasks, increasing both data efficiency and performance of the language model pre-training.

In this paper, we introduce a novel dIffusion language modEl pre-training framework for text generation, which we call GENIE. GENIE is a large-scale pre-trained diffusion language model that consists of an encoder and a diffusion-based decoder, which can generate text by gradually transforming a random noise sequence into a coherent text sequence. To pre-train GENIE on a large-scale language corpus, we design a new continuous paragraph denoise objective, which encourages the diffusion-decoder to reconstruct a clean text paragraph from a corrupted version, while preserving the semantic and syntactic coherence. We evaluate GENIE on four downstream text generation benchmarks, namely XSum, CNN/DailyMail, Gigaword, and CommonGen. Our experimental results show that GENIE achieves comparable performance with the state-of-the-art autoregressive models on these benchmarks, and generates more diverse text samples. The code and models of GENIE are available at https://github.com/microsoft/ProphetNet/tree/master/GENIE.

Knowledge distillation is an effective way to transfer knowledge from a strong teacher to an efficient student model. Ideally, we expect the better the teacher is, the better the student performs. However, this expectation does not always come true. It is common that a strong teacher model results in a bad student via distillation due to the nonnegligible gap between teacher and student. To bridge the gap, we propose PROD, a PROgressive Distillation method, for dense retrieval. PROD consists of a teacher progressive distillation and a data progressive distillation to gradually improve the student. To alleviate catastrophic forgetting, we introduce a regularization term in each distillation process. We conduct extensive experiments on seven datasets including five widely-used publicly available benchmarks: MS MARCO Passage, TREC Passage 19, TREC Document 19, MS MARCO Document, and Natural Questions, as well as two industry datasets: Bing-Rel and Bing-Ads. PROD achieves the state-of-the-art in the distillation methods for dense retrieval. Our 6-layer student model even surpasses most of the existing 12-layer models on all five public benchmarks. The code and models are released in https://github.com/microsoft/SimXNS.

Most existing pre-trained language representation models (PLMs) are sub-optimal in sentiment analysis tasks, as they capture the sentiment information from word-level while underconsidering sentence-level information.In this paper, we propose SentiWSP, a novel Sentiment-aware pre-trained language model with combined Word-level and Sentence-level Pre-training tasks.The word level pre-training task detects replaced sentiment words, via a generator-discriminator framework, to enhance the PLM's knowledge about sentiment words.The sentence level pre-training task further strengthens the discriminator via a contrastive learning framework, with similar sentences as negative samples, to encode sentiments in a sentence.Extensive experimental results show that SentiWSP achieves new state-of-the-art performance on various sentence-level and aspectlevel sentiment classification benchmarks.We have made our code and model publicly available at https://github.com/XMUDM/SentiWSP.

In recent years, data security incidents caused by insider threats in distributed file systems have attracted the attention of academia and industry. The most common way to detect insider threats is based on user profiles. Through analysis, we realize that based on existing user profiles are not efficient enough, and there are many false positives when a stable user profile has not yet been formed. In this work, we propose personalized user profiles and design an insider threat detection framework, which can intelligently detect insider threats for securing distributed file systems in real-time. To generate personalized user profiles, we come up with a time window-based clustering algorithm and a weighted kernel density estimation algorithm. Compared with non-personalized user profiles, both the Recall and Precision of insider threat detection based on personalized user profiles have been improved, resulting in their harmonic mean F1 increased to 96.52%. Meanwhile, to reduce the false positives of insider threat detection, we put forward operation recommendations based on user similarity to predict new operations that users will produce in the future, which can reduce the false positive rate (FPR). The FPR is reduced to 1.54% and the false positive identification rate (FPIR) is as high as 92.62%. Furthermore, to mitigate the risks caused by inaccurate authorization for users, we present user tags based on operation content and permission. The experimental results show that our proposed framework can detect insider threats more effectively and precisely, with lower FPR and high FPIR.