As LLMs continuously evolve, there is an urgent need for a reliable evaluation method that delivers trustworthy results promptly.Currently, static benchmarks suffer from inflexibility and unreliability, leading users to prefer human voting platforms like Chatbot Arena.However, human evaluations require significant manual effort.Therefore, we propose Auto-Arena, an innovative framework that automates the entire evaluation process using LLM-powered agents.Firstly, an LLM examiner generates questions.Then, two LLM candidates engage in a multi-round peer battle based on the questions, aiming at revealing their true performance differences.Finally, a committee of LLM judges collaboratively discusses and decides the winner, reducing bias and enhancing fairness.During the peer battles, we observe intriguing scenarios where the LLM candidates display competitive behaviors and learn from the opponents.In our extensive experiments involving 15 recent LLMs, Auto-Arena shows a 92.14% correlation with human preferences, surpassing all previous expert-annotated benchmarks without any manual efforts.Auto-Arena offers a promising alternative to current human evaluation platforms for evaluating LLMs automatically. 1

Yew Ken Chia, Liying Cheng, Hou Pong Chan, Maojia Song, Chaoqun Liu, Mahani Aljunied, Soujanya Poria, Lidong Bing. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

Deepanway Ghosal, Vernon Toh, Yew Ken Chia, Soujanya Poria. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

We present chain-of-knowledge (CoK), a novel framework that augments large language models (LLMs) by dynamically incorporating grounding information from heterogeneous sources. It results in more factual rationales and reduced hallucination in generation. Specifically, CoK consists of three stages: reasoning preparation, dynamic knowledge adapting, and answer consolidation. Given a knowledge-intensive question, CoK first prepares several preliminary rationales and answers while identifying the relevant knowledge domains. If there is no majority consensus among the answers from samples, CoK corrects the rationales step by step by adapting knowledge from the identified domains. These corrected rationales can plausibly serve as a better foundation for the final answer consolidation. Unlike prior studies that primarily use unstructured data, CoK also leverages structured knowledge sources such as Wikidata and tables that provide more reliable factual information. To access both unstructured and structured knowledge sources in the dynamic knowledge adapting stage, we propose an adaptive query generator that allows the generation of queries for various types of query languages, including SPARQL, SQL, and natural sentences. Moreover, to minimize error propagation between rationales, CoK corrects the rationales progressively using preceding corrected rationales to generate and correct subsequent rationales. Extensive experiments show that CoK consistently improves the performance of LLMs on knowledge-intensive tasks across different domains.

Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Boyang Li, Shafiq Joty, Lidong Bing. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

Despite the existence of various benchmarks for evaluating natural language processing models, we argue that human exams are a more suitable means of evaluating general intelligence for large language models (LLMs), as they inherently demand a much wider range of abilities such as language understanding, domain knowledge, and problem-solving skills. To this end, we introduce M3Exam, a novel benchmark sourced from real and official human exam questions for evaluating LLMs in a multilingual, multimodal, and multilevel context. M3Exam exhibits three unique characteristics: (1) multilingualism, encompassing questions from multiple countries that require strong multilingual proficiency and cultural knowledge; (2) multimodality, accounting for the multimodal nature of many exam questions to test the model's multimodal understanding capability; and (3) multilevel structure, featuring exams from three critical educational periods to comprehensively assess a model's proficiency at different levels. In total, M3Exam contains 12,317 questions in 9 diverse languages with three educational levels, where about 23\% of the questions require processing images for successful solving. We assess the performance of top-performing LLMs on M3Exam and find that current models, including GPT-4, still struggle with multilingual text, particularly in low-resource and non-Latin script languages. Multimodal LLMs also perform poorly with complex multimodal questions. We believe that M3Exam can be a valuable resource for comprehensively evaluating LLMs by examining their multilingual and multimodal abilities and tracking their development. Data and evaluation code is available at \url{https://github.com/DAMO-NLP-SG/M3Exam}.

Relation extraction has the potential for largescale knowledge graph construction, but current methods do not consider the qualifier attributes for each relation triplet, such as time, quantity or location.The qualifiers form hyperrelational facts which better capture the rich and complex knowledge graph structure.For example, the relation triplet (Leonard Parker, Educated At, Harvard University) can be factually enriched by including the qualifier (End Time, 1967).Hence, we propose the task of hyper-relational extraction to extract more specific and complete facts from text.To support the task, we construct HyperRED, a large-scale and general-purpose dataset.Existing models cannot perform hyper-relational extraction as it requires a model to consider the interaction between three entities.Hence, we propose Cu-beRE, a cube-filling model inspired by tablefilling approaches and explicitly considers the interaction between relation triplets and qualifiers.To improve model scalability and reduce negative class imbalance, we further propose a cube-pruning method.Our experiments show that CubeRE outperforms strong baselines and reveal possible directions for future research.Our code and data are available at github.com/declare-lab/HyperRED.

Lu Xu, Yew Ken Chia, Lidong Bing. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.