The Model Context Protocol (MCP) is rapidly emerging as a pivotal open standard, designed to enhance agent-tool integration and interoperability, and is positioned to unlock a new era of powerful, interconnected, and genuinely utilitarian agentic AI. However, despite MCP's growing adoption, existing benchmarks often fail to capture real-world agent performance within this new paradigm, leading to a distorted perception of their true operational value and an inability to reliably differentiate proficiencies. To bridge this critical evaluation gap, we introduce MCP-AgentBench—a comprehensive benchmark specifically engineered to rigorously assess language agent capabilities in MCP-mediated tool interactions. Core contributions of MCP-AgentBench include: the establishment of a robust MCP testbed comprising 33 operational servers with 188 distinct tools; the development of a benchmark featuring 600 systematically designed queries distributed across 6 distinct categories of varying interaction complexity; and the introduction of MCP-Eval, a novel outcome-oriented evaluation methodology prioritizing real-world task success. Through extensive empirical evaluation of leading language agents, we provide foundational insights. MCP-AgentBench aims to equip the research community with a standardized and reliable framework to build, validate, and advance agents capable of fully leveraging MCP's transformative benefits, thereby accelerating progress toward truly capable and interoperable AI systems.

Chiwei Zhu, Benfeng Xu, Mingxuan Du, Shaohan Wang, Xiaorui Wang, Zhendong Mao, Yongdong Zhang. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Web browsing—widely used for information retrieval and fact verification—has become a fundamental capability of recently emerged large language model (LLM) agents, which is often elicited by training on complex questions requiring web search. However, this task faces challenges with respect to data and training: existing QA datasets are mostly 1-3 hop over closed corpora (e.g., Wikipedia); meanwhile, outcome-based on-policy RL that used by recent works is inefficient and brittle in long-horizon, tool-heavy browsing environments. To address these challenges, we introduce GraphSynthQA, a knowledge-graph (KG)—guided synthesis framework in an open-web setting. Starting from Wikidata seed entities, GraphSynthQA iteratively retrieves and verifies evidence from the internet to expand a KG, then synthesizes complex, answer-verifiable queries grounded in multi-evidence dependencies. Building on the synthesized data, we train web-browsing agents with a compute-efficient two-stage recipe: (i) cold-start supervised fine-tuning on ReAct-style trajectories, and (ii) step-level Direct Preference Optimization (DPO), where preferences are constructed offline via single-step branched rollouts that contrast candidate actions by their downstream success rates, providing dense process supervision without expensive on-policy exploration. Experiments show that our approach consistently improves performance on challenging web-browsing benchmarks and remains competitive among models of similar size.

The pursuit of diverse, complex, and largescale instruction data is crucial for automatically aligning large language models (LLMs).While there are methods capable of generating synthetic instructions at scale, they either suffer from limited grounding sources, leading to a narrow distribution, or rely on trivial extensions that fail to produce meaningful trajectories in terms of complexity.In contrast, instructions that benefit efficient alignment are typically crafted with cognitive insights and grounded in real-world use cases.In this paper, we synthesize such instructions using attributed grounding, which involves 1) a top-down attribution process that grounds a selective set of real instructions to situated users, and 2) a bottom-up synthesis process that leverages web documents to first generate a situation, then a meaningful instruction.This framework allows us to harvest diverse and complex instructions at scale, utilizing the vast range of web documents.Specifically, we construct a dataset of 1 million instructions, called SYNTHQUESTIONS, and demonstrate that models trained on it achieve leading performance on several common benchmarks, with improvements that continually scale with more web corpora.Data, models and codes will be available at https://github. com/Ignoramus0817/SynthQuestions.

Open-domain Question Answering (ODQA) has long been an NLP task receiving wide attention of researchers. Despite being utilized various domains and applications, the evaluation of ODQA systems remains a complicated problem, which is worsened by the wide usage of large language models(LLMs). As LLMs often generate free-form answers that do not follow certain format, traditional string-matching-driven evaluation metrics like Lexical Match are not feasible to accurately reflect the performance of LLM-based ODQA systems. In the meantime, LLM-as-a-Judge methods with simple prompts also display limited consistency with human annotators. To tackle above challenges, we propose a framework of developing effective evaluation prompts based on iterative test and optimization, which can be conducted either by human or LLMs. The resulting prompt, which we refer to as PromptMetric, shows considerable advantages over traditional evaluation methods and LLM evaluation methods with basic prompts. We also demonstrate the robustness of our methods on different models, and show that PromptMetric can be highly economical when applied in ODQA evaluation.

Supervised fine-tuning (SFT) is a critical procedure for aligning large language models.Despite its efficiency, the construction of SFT data often struggles with issues of quality, diversity, and scalability.Many existing methods, inspired by the SELF-INSTRUCT framework, typically generate synthetic instructions by prompting aligned proprietary models like ChatGPT.However, such process suffers from stale distribution, resulting in instructions that are merely trivial variations of existing ones.In this paper, we introduce a novel bootstrapping approach termed KNN-INSTRUCT, which incorporates KNN deduction to produce meaningful new instructions by effectively summarizing and learning from similar existing ones.We conduct an economical controlled experiment to preliminarily validate its effectiveness.In the further experiment, we construct a high-quality SFT dataset named KNN-INST-12K*.Applying the dataset to Qwen-2-7B, we get a MT-Bench score of 7.64, which outperforms all 7B models on the LMSYS leaderboard, including Starling-LM-7B (7.48), OpenChat-3.5 (7.06) and Zephyr-7B-beta (6.53).Our code and data are available at https://github.com/ CrossmodalGroup/KNN-Instruct/.