arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.19073cs.CE
收藏

金融大语言模型与代理的评估与基准测试套件

Evaluation and Benchmarking Suite for Financial Large Language Models and Agents

Columbia University(哥伦比亚大学);Carnegie Mellon University(卡内基梅隆大学);Rensselaer Polytechnic Institute(伦斯勒理工学院);Boston College(波士顿学院);Stevens Institute of Technology(史蒂文斯理工学院);The FinAI;Red Hat(红帽);PyTorch Foundation(PyTorch基金会);Linux Foundation(Linux基金会)

Shengyuan Lin, Kaiwen He, Jaisal Patel, Qinchuan Zhang, Chris Ding, James Tang, Keyi Wang, Yupeng Cao, Yan Wang, Kairong Xiao, Vincent Caldeira, Matt White, Xiao-Yang Liu Yanglet

AI总结:

本文提出了一套评估和基准测试套件,用于评估和比较金融大语言模型与代理,促进更稳健的FinAI生态系统。

摘要:

在过去三年中,金融服务行业见证了大型语言模型(LLMs)和代理从探索阶段过渡到准备阶段和治理阶段。金融大语言模型(FinLLMs),如开源的FinGPT和专有的BloombergGPT,具有在金融应用中的巨大潜力,包括检索实时数据、辅导、分析社交媒体情绪、分析SEC文件以及代理交易。然而,通用目的LLMs和代理缺乏金融专业知识,往往难以处理复杂的金融推理。本文提出了一套评估和基准测试套件,涵盖FinLLMs和FinAgents的整个生命周期。该套件由SecureFinAI实验室领导,包括一个评估流程和与Linux基金会和PyTorch基金会合作的治理框架,一个与HuggingFace合作的FinLLM排行榜,一个与Red Hat合作的AgentOps框架,以及一个与Rensselear开放源研究中心合作的文档网站。我们的协同开发经历三个阶段:FinLLM探索(2023)、FinLLM准备(2024)和FinAI治理(2025)。所提出的套件是一个开放平台,使研究人员和从业者能够对不同的FinLLMs和FinAgents进行定量和定性分析,促进更加稳健和可靠的FinAI生态系统。

英文摘要:

Over the past three years, the financial services industry has witnessed Large Language Models (LLMs) and agents transitioning from the exploration stage to readiness and governance stages. Financial large language models (FinLLMs), such as open FinGPT and proprietary BloombergGPT , have great potential in financial applications, including retrieving real-time data, tutoring, analyzing sentiment of social media, analyzing SEC filings, and agentic trading. However, general-purpose LLMs and agents lack financial expertise and often struggle to handle complex financial reasoning. This paper presents an evaluation and benchmarking suite that covers the lifecycle of FinLLMs and FinAgents. This suite led by SecureFinAI Lab includes an evaluation pipeline and a governance framework collaborating with Linux Foundation and PyTorch Foundation, a FinLLM Leaderboard with HuggingFace, an AgentOps framework with Red Hat, and a documentation website with Rensselear Center of Open Source. Our collaborative development evolves through three stages: FinLLM Exploration (2023), FinLLM Readiness (2024), and FinAI Governance (2025). The proposed suite serves as an open platform that enables researchers and practitioners to perform both quantitative and qualitative analysis of different FinLLMs and FinAgents, fostering a more robust and reliable FinAI ecosystem.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑