Recent studies have demonstrated that large language models (LLMs) are susceptible to being misled by false premise questions (FPQs), leading to errors in factual knowledge, known as factuality hallucination. Existing benchmarks that assess this vulnerability primarily rely on manual construction, resulting in limited size and lack of expandability. In this work, we introduce an automated, scalable pipeline to create FPQs based on knowledge graphs (KGs). The first step is to modify true triplets extracted from KGs to create false premises. Subsequently, utilizing the state-of-the-art capabilities of GPTs, we generate semantically rich FPQs. Based on the proposed method, we present a comprehensive benchmark, the Knowledge Graph-based False Premise Questions (KG-FPQ), which contains approximately 178k FPQs across three knowledge domains, at six levels of confusability, and in two task formats. Using KG-FPQ, we conduct extensive evaluations on several representative LLMs and provide valuable insights. The KG-FPQ dataset and code are available at https://github.com/yanxuzhu/KG-FPQ.

In recent years, large language models (LLMs) have made significant progress in code generation. However, as these models are increasingly adopted for software development, their associated security risks have become more pronounced. Studies have shown that traditional deep learning robustness issues also adversely affect the reliability of code generation. In this paper, we use game theory to systematically examine security vulnerabilities in code generation and illustrate how attackers can propagate malicious models to create genuine threats. We also demonstrate, for the first time, that attackers can leverage user behavior as a trigger for backdoor attacks—dynamically controlling when malicious code is injected—and calibrate these attacks to a user’s skill level, leading to varying degrees of impact. Through extensive experiments on leading code generation models, we verify that these security threats are both feasible and dangerous. Our research code will be available at https://github.com/KirinNg/Adaptive_Malicious_Code_Injection_Backdoor_Attack.