Competency Questions (CQs) play a crucial role in validating ontology design. While manually crafting CQs can be highly time-consuming and costly for ontology engineers, recent studies have explored the use of large language models (LLMs) to automate this process. However, prior approaches have largely evaluated generated CQs based on their similarity to existing datasets, which often fail to verify semantic pitfalls such as “Misusing allValuesFrom”. Since such pitfalls cannot be reliably detected through rule-based methods, we propose a novel dataset and model of Validating Semantic Pitfalls in Ontology (VSPO) for CQ generation specifically designed to verify the semantic pitfalls. To simulate missing and misused axioms, we use LLM to generate natural language definitions of classes and properties and introduce misalignments between the definitions and the ontology by removing axioms or altering logical operators (e.g., substituting union with intersection). We then fine-tune LLaMA-3.1-8B-Instruct to generate CQs that validate these semantic discrepancies between the provided definitions and the corresponding axioms. The resulting CQs can detect a broader range of modeling errors compared to existing public datasets. Our fine-tuned model demonstrates superior performance over baselines, showing 26% higher precision and 28.2% higher recall than GPT-4.1 in generating CQs for pitfall validation. This research enables automatic generation of TBox-validating CQs using LLMs, significantly reducing manual effort while improving semantic alignment between ontologies and expert knowledge. To the best of our knowledge, this is the first study to target semantic pitfall validation in CQ generation using LLMs.

Knowledge graphs (KGs) provide a structured representation of real-world facts as triples consisting of entities and their relationships.With the rapid progress of large language models (LLMs), recent studies increasingly explore LLMs for end-to-end KG construction from text.In particular, generative knowledge extraction (GKE) builds KGs by directly generating structured triples from documents.However, generation errors are inevitable, and the resulting KGs often contain triples that do not align with the facts expressed in the source text.To address these issues, we propose GraphRefine, a framework that performs triple-level refinement on KGs constructed via GKE.We first analyze factual inconsistencies that arise in GKE and categorize their types based on a human evaluation.We then construct training data reflecting these types and fine-tune an LLM as a KG refiner.Given a draft KG, the fine-tuned refiner selects a refinement operation for each triple and, if needed, deletes, edits, or rewrites it to reduce factual inconsistencies.Extensive experiments demonstrate that GraphRefine goes beyond deletion-only approaches and improves KG quality from diverse perspectives.Table 8: Qualitative comparison of knowledge extraction results (Draft KG vs. GraphJudge vs. GraphRefine).