arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.05710q-fin.CPcs.AI
收藏

FinReflectKG -- EvalBench:基于多维评估的金融知识图谱基准测试

FinReflectKG -- EvalBench: Benchmarking Financial KG with Multi-Dimensional Evaluation

New York, US(美国纽约);Gurugram, India(印度古尔冈)

Fabrizio Dimino, Abhinav Arun, Bhaskarjit Sarmah, Stefano Pasquali

AI总结:

本文提出FinReflectKG-EvalBench,通过多维评估框架对金融知识图谱提取进行基准测试,证明基于反思的提取方法在全面性、精度和相关性上表现最佳,同时验证LLM作为评判者在成本效率和结构化错误分析中的可靠性。

摘要:

大型语言模型(LLMs)越来越多地被用于从非结构化金融文本中提取结构化知识。尽管先前研究已探索了各种提取方法,但目前尚无统一的基准或评估框架用于构建金融知识图谱(KG)。我们介绍了FinReflectKG-EvalBench,这是一个用于从SEC 10-K文件中提取KG的基准和评估框架。基于FinReflectKG的代理和整体评估原则,该框架将审计三元组链接到S&P 100文件的源片段,并支持单次通过、多次通过和基于反思代理的提取模式。EvalBench实现了确定性的提交-然后-辩护评判协议,具有明确的偏见控制,以缓解位置效应、宽容度、冗余性和世界知识依赖性。每个候选三元组都通过二元判断的忠实性、精度和相关性进行评估,而全面性则在片段层面通过三级顺序量表(好、部分、坏)进行评估。我们的发现表明,当配备明确的偏见控制时,LLM作为评判者协议提供了一种可靠且成本效益高的替代方案,同时也能实现结构化的错误分析。基于反思的提取方法在全面性、精度和相关性方面表现最佳,而单次通过提取则在忠实性上保持最高水平。通过聚合这些互补的维度,FinReflectKG-EvalBench实现了细粒度的基准测试和偏见意识的评估,推动了金融AI应用的透明度和治理。

英文摘要:

Large language models (LLMs) are increasingly being used to extract structured knowledge from unstructured financial text. Although prior studies have explored various extraction methods, there is no universal benchmark or unified evaluation framework for the construction of financial knowledge graphs (KG). We introduce FinReflectKG - EvalBench, a benchmark and evaluation framework for KG extraction from SEC 10-K filings. Building on the agentic and holistic evaluation principles of FinReflectKG - a financial KG linking audited triples to source chunks from S&P 100 filings and supporting single-pass, multi-pass, and reflection-agent-based extraction modes - EvalBench implements a deterministic commit-then-justify judging protocol with explicit bias controls, mitigating position effects, leniency, verbosity and world-knowledge reliance. Each candidate triple is evaluated with binary judgments of faithfulness, precision, and relevance, while comprehensiveness is assessed on a three-level ordinal scale (good, partial, bad) at the chunk level. Our findings suggest that, when equipped with explicit bias controls, LLM-as-Judge protocols provide a reliable and cost-efficient alternative to human annotation, while also enabling structured error analysis. Reflection-based extraction emerges as the superior approach, achieving best performance in comprehensiveness, precision, and relevance, while single-pass extraction maintains the highest faithfulness. By aggregating these complementary dimensions, FinReflectKG - EvalBench enables fine-grained benchmarking and bias-aware evaluation, advancing transparency and governance in financial AI applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑