arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2607.01740cs.AI
收藏

金融服务大语言模型评估的元基准

Meta-Benchmarks for Financial-Services LLM Evaluation

Commonwealth Bank of Australia(澳大利亚联邦银行)

Blair Hudson

AI总结:

针对公共排行榜忽略金融服务特定认知需求的问题,提出元基准框架,将452个基准映射到41个O*NET工作活动和38个BIAN银行业务领域,采用乘法加权方案和Elo锦标赛生成可比较的工作活动与业务领域得分。

摘要:

公共大语言模型排行榜优化全局平均性能,未能捕捉金融服务工作的特定认知需求:在MMLU-Pro上领先的模型可能在基于文档的合规推理中表现不佳,而编程领先的模型可能难以处理多轮客户交互。我们提出一个元基准框架,将452个公开报告的基准组织成41个O*NET通用工作活动,并将这些活动聚合到38个BIAN银行业务领域,涵盖销售、运营、风险和支持工作。一个乘法加权方案(区分度×覆盖率×时效性),在滚动模型窗口上计算,奖励那些仍能区分最佳模型、被广泛报告且仍在活跃使用的基准,自动抑制饱和的遗留测试。这些权重缩放成对Elo锦标赛中的K因子,产生跨基准可比的工作活动得分,无需原始分数归一化;业务领域得分是组成工作活动Elo的加权平均值。我们在一个截至2026年6月、涵盖25个组织288个模型的时点公开快照上演示了该框架,并描述了方法论、完整分类、设计决策和局限性,旨在使该方法对面临类似选择和治理挑战的机构可复现。

英文摘要:

Public LLM leaderboards optimise for global average performance and do not capture the specific cognitive demands of financial-services work: a model that leads on MMLU-Pro may underperform on document-grounded compliance reasoning, and a coding leader may handle multi-turn customer interactions poorly. We present a meta-benchmarking framework that organises 452 publicly reported benchmarks into 41 O*NET Generalized Work Activities and aggregates those into 38 BIAN banking business domains spanning sales, operations, risk, and support work. A multiplicative weighting scheme (discrimination x coverage x recency), computed over a rolling model window, rewards benchmarks that still separate the best models, are widely reported, and remain in active use, suppressing saturated legacy tests automatically. These weights scale the K-factor in a pairwise Elo tournament, producing cross-benchmark-comparable work-activity scores without raw score normalisation; business-domain scores are weighted averages of the constituent work-activity Elos. We demonstrate the framework on a point-in-time public snapshot covering 288 models across 25 organisations as of June 2026, and describe the methodology, full taxonomy, design decisions, and limitations with the aim of making the approach reproducible for institutions facing similar selection and governance challenges.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑