Yakun Zhu, Yutong Huang, Shengqian Qin, Zhongzhen Huang, Shaoting Zhang, Xiaofan Zhang. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Large language models (LLMs) have showcased remarkable capabilities as autonomous agents when augmented with external tools.Equipped with fixed tool sets, LLMs struggle with addressing diverse user inquiries in open-world tasks.To evaluate and boost the performance of LLMs in dealing with complex demands in the real-world, we propose open-world function calling, where LLMs need to retrieve suitable tools from a pre-defined external tool library and use retrieved tools to resolve the user's problem.We introduce Meta-Tool, a versatile and plug-and-play tool retrieval system as the access of LLMs to external tool library.Drawing inspiration from the myriad of enhanced approaches associated with Retrieval-Augmented Generation (RAG), Meta-Tool employs a hypothesizeretrieve-invoke framework.We further propose Meta-Bench, a comprehensive benchmark for evaluating LLMs in open-world function calling and associated tasks.Meta-Bench encompasses 2, 800 dialogues and 7, 361 tools, spanning ten distinct scenarios to provide robust and diverse test categories.In conjunction, we present MT-LLaMA, a finetuned version of LLaMA-3.1, which exhibits remarkable performance improvements.Our empirical experiments reveal that Meta-Tool significantly enhances the ability of advanced LLMs to retrieve and leverage the most suitable tools compared to previous tool retrieval methods.Moreover, our fine-tuning enables even smallersized LLMs to achieve comparable even exceeding results to GPT-4o.Both the benchmark and the model are made publicly available at https://github.com/qinshengqian/Meta-Tool to foster further research and development in the field.