arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.24943cs.AIcs.CL
收藏

FinMCP-Bench:基于模型上下文协议的LLM代理在真实金融工具使用中的基准测试

FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under the Model Context Protocol

Qwen DianJin Team, Alibaba Cloud Computing(阿里云云计算千问点睛团队);YINGMI Wealth Management(盈米财富管理);Soochow University(苏州大学)

Jie Zhu, Yimin Tian, Boyang Li, Kehao Wu, Zhongzhi Liang, Junhui Li, Xianyin Zhang, Lifan Guo, Feng Chen, Yong Liu, Chi Zhang

AI总结:

本文提出FinMCP-Bench,用于评估LLM在解决真实金融问题时通过调用金融模型上下文协议的能力,包含613个样本,涵盖10种主要场景和33种子场景,提供真实和合成查询,评估不同任务复杂度下的模型表现。

摘要:

本文介绍了FinMCP-Bench,一种新的基准,用于评估大型语言模型(LLM)通过调用金融模型上下文协议来解决真实金融问题的能力。FinMCP-Bench包含613个样本,涵盖10种主要场景和33种子场景,包含真实和合成用户查询,以确保多样性和真实性。它整合了65个真实金融MCPs和三种样本类型:单工具、多工具和多轮对话,允许在不同任务复杂性层面评估模型。利用此基准,我们系统地评估了主流LLM,并提出了明确衡量工具调用准确性和推理能力的指标。FinMCP-Bench为推进金融LLM代理研究提供了标准化、实用且具有挑战性的测试平台。

英文摘要:

This paper introduces \textbf{FinMCP-Bench}, a novel benchmark for evaluating large language models (LLMs) in solving real-world financial problems through tool invocation of financial model context protocols. FinMCP-Bench contains 613 samples spanning 10 main scenarios and 33 sub-scenarios, featuring both real and synthetic user queries to ensure diversity and authenticity. It incorporates 65 real financial MCPs and three types of samples, single tool, multi-tool, and multi-turn, allowing evaluation of models across different levels of task complexity. Using this benchmark, we systematically assess a range of mainstream LLMs and propose metrics that explicitly measure tool invocation accuracy and reasoning capabilities. FinMCP-Bench provides a standardized, practical, and challenging testbed for advancing research on financial LLM agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑