Evaluation and Benchmarking Suite for Financial Large Language Models and Agents
Columbia University(哥伦比亚大学);Carnegie Mellon University(卡内基梅隆大学);Rensselaer Polytechnic Institute(伦斯勒理工学院);Boston College(波士顿学院);Stevens Institute of Technology(史蒂文斯理工学院);The FinAI;Red Hat(红帽);PyTorch Foundation(PyTorch基金会);Linux Foundation(Linux基金会)
Shengyuan Lin, Kaiwen He, Jaisal Patel, Qinchuan Zhang, Chris Ding, James Tang, Keyi Wang, Yupeng Cao, Yan Wang, Kairong Xiao, Vincent Caldeira, Matt White, Xiao-Yang Liu Yanglet
Over the past three years, the financial services industry has witnessed Large Language Models (LLMs) and agents transitioning from the exploration stage to readiness and governance stages. Financial large language models (FinLLMs), such as open FinGPT and proprietary BloombergGPT , have great potential in financial applications, including retrieving real-time data, tutoring, analyzing sentiment of social media, analyzing SEC filings, and agentic trading. However, general-purpose LLMs and agents lack financial expertise and often struggle to handle complex financial reasoning. This paper presents an evaluation and benchmarking suite that covers the lifecycle of FinLLMs and FinAgents. This suite led by SecureFinAI Lab includes an evaluation pipeline and a governance framework collaborating with Linux Foundation and PyTorch Foundation, a FinLLM Leaderboard with HuggingFace, an AgentOps framework with Red Hat, and a documentation website with Rensselear Center of Open Source. Our collaborative development evolves through three stages: FinLLM Exploration (2023), FinLLM Readiness (2024), and FinAI Governance (2025). The proposed suite serves as an open platform that enables researchers and practitioners to perform both quantitative and qualitative analysis of different FinLLMs and FinAgents, fostering a more robust and reliable FinAI ecosystem.
FinBoardBench: Benchmarking Dynamic Wealth Management and Strategic Financial Reasoning of LLMs via Board Game Simulations
School of Computer Science and Engineering, Macau University of Science and Technology, Macau, China(1 计算机科学与工程学院,澳门科技大学,澳门,中国);School of Economics, Anhui University, Anhui, China(2 经济学院,安徽大学,安徽,中国);SKLPlanets, Macau University of Science and Technology, Macau, China(3 SKLPlanets,澳门科技大学,澳门,中国);Department of Computer and Information Science, University of Macau, Macau, China(4 计算机与信息科学系,澳门大学,澳门,中国);School of Energy and Power Engineering, Huazhong University of Science and Technology, Hubei, China(5 能源与动力工程学院,华中科技大学,湖北,中国)
Recognition Is Not Grounding: Benchmarking Multilingual Financial Evidence Extraction
The Fin AI;University of Manchester(曼彻斯特大学);Georgia Institute of Technology(佐治亚理工学院);New Jersey Institute of Technology(新泽西理工学院);Cardiff University(卡迪夫大学)
FinTagging: Benchmarking LLMs for Extracting and Structuring Financial Information
Georgia Institute of Technology(佐治亚理工学院);Columbia University(哥伦比亚大学);California State University(加州州立大学);University of Montreal(蒙特利尔大学);Carnegie Mellon University(卡内基梅隆大学);Rensselaer Polytechnic Institute(莱斯利理工学院);The University of Manchester(曼彻斯特大学);Harvard University(哈佛大学)
Representation Learning of Limit Order Book: A Comprehensive Study and Benchmarking
Harbin Institute of Technology(哈尔滨工业大学);Guangdong Key Laboratory of Brain-Inspired Intelligent Computation(广东省脑启发智能计算重点实验室);Department of Computer Science and Engineering(计算机科学与工程系);Southern University of Science and Technology(南方科技大学);Department of Statistics and Data Science(统计与数据科学系)