Large Language Models (LLMs) have witnessed remarkable advancements in recent years, prompting the exploration of tool learning, which integrates LLMs with external tools to address diverse real-world challenges. Assessing the capability of LLMs to utilise tools necessitates large-scale and stable benchmarks. However, previous works relied on either hand-crafted online tools with limited scale, or large-scale real online APIs suffering from instability of API status. To address this problem, we introduce StableToolBench, a benchmark evolving from ToolBench, proposing a virtual API server and stable evaluation system. The virtual API server contains a caching system and API simulators which are complementary to alleviate the change in API status. Meanwhile, the stable evaluation system designs solvable pass and win rates using GPT-4 as the automatic evaluator to eliminate the randomness during evaluation. Experimental results demonstrate the stability of StableToolBench, and further discuss the effectiveness of API simulators, the caching system, and the evaluator system.
StableToolBench-MirrorAPI: Modeling Tool Environments as Mirrors of 7,000+ Real-World APIs
Tsinghua University(清华大学);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院);RayNeo(雷尼奥);Google(谷歌);The University of Toronto(多伦多大学);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院)
Towards Comprehensive Stage-wise Benchmarking of Large Language Models in Fact-Checking
Department of Computer Science, Hong Kong Baptist University(香港 Baptist 大学计算机科学系);Salesforce AI Research(Salesforce AI 研究院);The Hong Kong University of Science and Technology(香港科学与技术大学);National University of Singapore(新加坡国立大学);The Education University of Hong Kong(香港教育大学)
CharacterBench: Benchmarking Character Customization of Large Language Models
Tsinghua University(清华大学);Lingxin AI(灵犀人工智能);University of Electronic Science and Technology of China(电子科技大学);Central South University(中南大学);Netease(网易)
Advancing Tool-Augmented Large Language Models via Meta-Verification and Reflection Learning
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China(认知智能国家重点实验室,中国科学技术大学);Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology(社会计算与交互机器人研究中心,哈尔滨工业大学)
Beyond Factual Knowledge: Benchmarking and Learning Step-Level Procedural Rule Reasoning in Large Language Models
University of Chinese Academy of Sciences(中国科学院大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Academy of Mathematics and Systems Science, Chinese Academy of Sciences(中国科学院数学与系统科学研究院)
BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law
Technical University of Munich (TUM)(慕尼黑技术大学);Ludwig Maximilian University of Munich (LMU)(慕尼黑路德维希-马克西米利安大学);University of Konstanz(康斯坦茨大学);University of Saarbrücken(萨尔布吕肯大学)
PRiSM: Benchmarking Phone Realization in Speech Models
CMU(卡内基梅隆大学);Gwangju Institute of Science and Technology(光州科学技术院);UT Austin(得克萨斯大学奥斯汀分校);LMU Munich(慕尼黑路德维希-马克西米利安大学);UC Berkeley(伯克利加州大学);NVIDIA(英伟达);UBC(不列颠哥伦比亚大学)