arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2504.04596cs.AIcs.CEcs.CL
收藏

SECQUE:一个用于评估现实世界金融分析能力的基准

SECQUE: A Benchmark for Evaluating Real-World Financial Analysis Capabilities

Microsoft Industry AI(微软产业人工智能)

Noga Ben Yoash, Meni Brief, Oded Ovadia, Gil Shenderovitz, Moshik Mishaeli, Rachel Lemberg, Eitam Sheetrit

AI总结:

本文提出SECQUE基准,包含565道专家编写的SEC文件分析问题,并开发SECQUE-Judge评估机制,用于系统衡量大语言模型的金融分析能力。

摘要:

我们引入了SECQUE,一个用于评估大语言模型(LLMs)在金融分析任务中表现的综合基准。SECQUE包含565道由专家撰写的问题,覆盖SEC文件分析的四个关键类别:比较分析、比率计算、风险评估和金融洞察生成。为了评估模型性能,我们开发了SECQUE-Judge,一种利用多个基于LLM的评判者的评估机制,该机制展现出与人类评估的高度一致性。此外,我们对各种模型在该基准上的表现进行了广泛分析。通过公开SECQUE,我们旨在促进金融人工智能领域的进一步研究和进步。

英文摘要:

We introduce SECQUE, a comprehensive benchmark for evaluating large language models (LLMs) in financial analysis tasks. SECQUE comprises 565 expert-written questions covering SEC filings analysis across four key categories: comparison analysis, ratio calculation, risk assessment, and financial insight generation. To assess model performance, we develop SECQUE-Judge, an evaluation mechanism leveraging multiple LLM-based judges, which demonstrates strong alignment with human evaluations. Additionally, we provide an extensive analysis of various models' performance on our benchmark. By making SECQUE publicly available, we aim to facilitate further research and advancements in financial AI.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑