arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.07294cs.CEcs.AI
收藏

Fin-RATE：面向SEC文件的金融分析与追踪评估基准

Fin-RATE: A Real-world Financial Analytics and Tracking Evaluation Benchmark for LLMs on SEC Filings

Tongji University（同济大学）；University of California, San Diego（加州大学圣地亚哥分校）；Yale University（耶鲁大学）；Goldman Sachs（高盛集团）

Yidong Jiang, Junrong Chen, Eftychia Makri, Jialin Chen, Peiwen Li, Ali Maatouk, Leandros Tassiulas, Eliot Brenner, Bing Xiang, Rex Ying

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对LLM在金融领域分析复杂监管文件的需求，提出基于SEC文件的Fin-RATE基准，通过三种任务路径评估模型，发现跨文档和跨时间分析时性能显著下降。

摘要：

随着大型语言模型（LLM）在金融领域的部署日益增多，LLM越来越需要解析复杂的监管披露文件。然而，现有基准通常关注孤立细节，未能反映专业分析的复杂性——这种分析需要综合多个文档、报告期和公司实体的信息。此外，这些基准无法区分错误源于检索失败、生成不准确、领域特定推理错误还是对查询或上下文的误解，从而难以精确诊断性能瓶颈。为弥补这些不足，我们引入Fin-RATE，这是一个基于美国证券交易委员会（SEC）文件构建的基准，通过三条路径模拟金融分析师的工作流程：单个披露文件内的细节导向推理、共享主题下的跨实体比较，以及同一公司在多个报告期内的纵向跟踪。我们在真实上下文和检索增强设置下，对17个领先的LLM（包括开源、闭源和金融专用模型）进行了基准测试。结果显示，当任务从单文档推理转向纵向和跨实体分析时，性能显著下降，准确率分别下降18.60%和14.35%。这种下降与比较幻觉、时间和实体不匹配的增加有关，并进一步反映在推理质量和事实一致性的下降上——这些局限性是现有基准尚未正式分类或量化的。

英文摘要：

With the increasing deployment of Large Language Models (LLMs) in the finance domain, LLMs are increasingly expected to parse complex regulatory disclosures. However, existing benchmarks often focus on isolated details, failing to reflect the complexity of professional analysis that requires synthesizing information across multiple documents, reporting periods, and corporate entities. Furthermore, these benchmarks do not disentangle whether errors arise from retrieval failures, generation inaccuracies, domain-specific reasoning mistakes, or misinterpretation of the query or context, making it difficult to precisely diagnose performance bottlenecks. To bridge these gaps, we introduce Fin-RATE, a benchmark built on U.S. Securities and Exchange Commission (SEC) filings and mirroring financial analyst workflows through three pathways: detail-oriented reasoning within individual disclosures, cross-entity comparison under shared topics, and longitudinal tracking of the same firm across reporting periods. We benchmark 17 leading LLMs, spanning open-source, closed-source, and finance-specialized models, under both ground-truth context and retrieval-augmented settings. Results show substantial performance degradation, with accuracy dropping by 18.60% and 14.35% as tasks shift from single-document reasoning to longitudinal and cross-entity analysis. This degradation is associated with increased comparison hallucinations, temporal and entity mismatches, and is further reflected in declines in reasoning quality and factual consistency--limitations that existing benchmarks have yet to formally categorize or quantify.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑