FinExam-10K：检索如何助力金融推理？

中文摘要 AI 辅助

专业金融考试要求模型结合领域知识、计算与判断，但尚无基准在同一协议下覆盖CFA（特许金融分析师）和FRM（金融风险管理师）的完整体系。我们推出FinExam-10K，据我们所知是该场景下已发布的最大规模英文基准，包含10198道经专家重新标注的题目，覆盖CFA一级至三级及FRM第一部分至第二部分。我们发布5110道题目，留存5088道用于季度维护的排行榜。为区分覆盖范围与局部答案可解答性，我们设置了含10198个题目的全覆盖赛道和含7625个题目的上下文完整推理赛道，后者是关于基于提供记录进行推理的主张的主要依据。在17个模型中，整体最佳准确率为85.29%；在冻结的困难分组中，全覆盖赛道最佳得分为34.68%，372道上下文完整题目上最佳得分为54.57%。17个模型共有47个上下文完整失败案例。Function-RAG和FunctionGraph-RAG可纠正数百个错误，但也推翻许多正确答案，产生极少或负的净增益。一个仅在公开数据上训练的门控模型，会根据问题和初始响应决定何时运行FunctionGraph-RAG。在5088道留存题目上，该门控模型对7.9%的题目调用FunctionGraph-RAG，使准确率从70.83%提升至71.23%（p=0.0446）。

英文摘要

Professional financial examinations require models to combine domain knowledge, calculation, and judgment, yet no benchmark covers the full CFA and FRM structure under one protocol. We introduce FinExam-10K, to our knowledge the largest reported English benchmark for this setting, with 10,198 expert-reannotated questions spanning CFA Levels I-III and FRM Parts I-II. We release 5,110 questions and sequester 5,088 for a quarterly maintained leaderboard. To separate coverage from local answerability, we report a 10,198-item Full-Coverage Track and a 7,625-item Context-Complete Reasoning Track, which is the primary basis for claims about reasoning from the supplied record. Across 17 models, the best accuracy is 85.29% overall. On the frozen Hard band, the best score is 34.68% on the Full-Coverage Track and 54.57% on the 372 context-complete items. All 17 models share 47 context-complete failures. Function-RAG and FunctionGraph-RAG rescue hundreds of errors but also overturn many correct answers, producing little or negative net gain. A gate trained only on public data decides from the question and initial response when FunctionGraph-RAG should run. On the 5,088 held-out items, the gate invokes FunctionGraph-RAG for 7.9% of questions and improves accuracy from 70.83% to 71.23% (p = .0446).

发表机构

• Newcastle University（纽卡斯尔大学）
• University of Manchester（曼彻斯特大学）
• University of Melbourne（墨尔本大学）
• University of Aberdeen（阿伯丁大学）
• University of Groningen（格罗宁根大学）
• MBZUAI（穆罕默德·本·扎耶德人工智能大学）
• INSAIT
• Sofia University “St. Kliment Ohridski”（索非亚大学“圣·克利门特·奥赫里德斯基”）

机构由 AI 辅助整理，请以论文原文为准。