面向可信财报电话会议记录分析的大语言模型引用基准

中文摘要 AI 辅助

大语言模型（LLMs）越来越多地被用于财务文档分析，包括财报电话会议记录（ECTs）。除了生成独立的陈述外，用户越来越倾向于基于引用的分析，即将陈述与源文档中可验证的引用配对，以便进行独立验证。然而，评估此类分析性陈述通常需要大量专家标注，这既昂贵又难以扩展，而且现实世界中的财务分析通常涉及长上下文的问题-答案三元组，进一步增加了任务复杂性。为了应对这些挑战并评估当前LLMs基于引用分析的现状，我们提出了一种数值证据评估方法，该方法无需依赖专家标注即可实现基于引用的评估。我们还引入了一个自动化数据集构建流程，并从标准普尔500指数的前100只成分股构建了ECTs-100数据集，以支持对基于引用性和正确性的基准测试。此外，我们考察了有意识的无能，这是财务分析中一种实际失败模式，即LLMs必须检测到可用证据不足并避免产生无支持的幻觉。实证结果表明，LLMs在基于引用性方面表现良好，但在正确性方面存在显著局限性，信息不足构成了额外挑战。

英文摘要

Large language models (LLMs) have been increasingly used for financial document analysis, including earnings call transcripts (ECTs). Beyond generating standalone claims, users increasingly prefer grounded analyses that pair claims with verifiable citations from source documents to enable independent validation. However, evaluating such analytical claims typically requires extensive expert annotation, which is costly and difficult to scale, and real-world financial analysis commonly involves long context-question-answer triplets, further increasing task complexity. To address these challenges and benchmark the current landscape of grounded analysis by LLMs, we propose a numeric evidence evaluation method that enables groundedness assessment without reliance on expert annotation. We also introduce an automated dataset construction pipeline and construct ECTs-100 from the top 100 constituents of the S&P 500 to support benchmark of both groundedness and correctness. In addition, we examine conscious incompetence, a practical failure mode in financial analysis in which LLMs must detect when available evidence is insufficient and refrain from producing unsupported hallucinations. Empirical results show that LLMs perform well in groundedness but face notable limitations in correctness, with informational insufficiency presenting an additional challenge.

发表机构

• American Express（美国运通）
• Singapore Decision Science Center of Excellence, American Express（美国运通新加坡决策科学卓越中心）
• Global Decision Science, American Express（美国运通全球决策科学部）

机构由 AI 辅助整理，请以论文原文为准。