arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.06401cs.AIcs.CL
收藏

BizFinBench.v2:通过真实用户数据和离线/在线双语评估实现金融领域可靠的大语言模型

BizFinBench.v2: Towards Reliable LLMs in Finance via Real-User Data and Offline/Online Bilingual Evaluation

HiThink Research(HiThink研究机构);Shanghai University of Finance and Economics(上海财经大学)

Xin Guo, Rongjunchen Zhang, Guilong Lu, Xuntao Guo, Shuai Jia, Zhi Yang, Liwen Zhang

AI总结:

研究针对大语言模型在金融应用中实际效果与报告性能差距大的问题,构建BizFinBench.v2基准,涵盖中美股票市场真实用户数据及多个任务,通过实验评估模型,揭示现有模型局限,为金融领域大语言模型部署提供基础。

摘要:

大语言模型在金融应用中日益重要,但现有基准很大程度依赖模拟或通用数据,导致实际效果与报告性能差距大。为此,我们提出BizFinBench.v2,首个基于中美股票市场真实用户问答数据构建的离线和在线集成基准,含8个离线和2个在线任务共28860个问题。实验表明GPT-5准确率仅61.5%,未达实际业务要求(84.8%),商业模型中DeepSeek-R1投资效果更佳。基于实际金融实践的错误分析揭示现有模型存在持续局限。BizFinBench.v2克服了先前基准的限制,为推进金融领域大语言模型部署提供了坚实基础。

英文摘要:

Large language models are becoming increasingly significant in financial applications. Nevertheless, prevailing benchmarks are largely dependent on simulated or generic data, which leads to a significant gap between reported performance and actual efficacy in real-world scenarios. To tackle this challenge, we present BizFinBench.v2, the first integrated offline and online benchmark built upon authentic user query-response data from both Chinese and U.S. equity markets. It comprises 28,860 questions across eight offline and two online tasks. Experimental results show that GPT-5 achieves a mere 61.5% accuracy, still failing to meet the practical business requirement (84.8%). Among the evaluated commercial models, DeepSeek-R1 exhibits superior investment efficacy. Error analysis grounded in real financial practice reveals persistent limitations in existing models. By overcoming the constraints of prior benchmarks, BizFinBench.v2 provides a substantiated foundation for advancing LLM deployment in the financial sector. Our data and code are available at https://github.com/HiThink-Research/BizFinBench.v2.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑