arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.10748cs.DB
收藏

对大型语言模型在知识图谱验证中的基准测试

Benchmarking Large Language Models for Knowledge Graph Validation

Aalto University(阿尔托大学);University of Padua(帕多瓦大学)

Farzad Shami, Stefano Marchesin, Gianmaria Silvello

AI总结:

本文提出FactCheck基准,评估大型语言模型在知识图谱验证中的性能,发现其在稳定性、可靠性及多模型共识策略上的不足,强调了建立系统评估框架的重要性。

摘要:

知识图谱(KGs)通过实体之间的关系来存储结构化的事实性知识,这对许多应用至关重要。这些应用依赖于KG的事实准确性,因此验证事实至关重要,但具有挑战性。专家手动验证是理想的选择,但在大规模上却不切实际。自动化方法显示出潜力,但尚未准备好用于现实中的KG。大型语言模型(LLMs)因其语义理解和知识访问能力而具有潜力,但其在KG事实验证中的适用性和有效性仍 largely 未被探索。在本文中,我们介绍了FactCheck,一个旨在评估LLMs在KG事实验证上的基准,涵盖三个关键维度:(1)LLMs内部知识;(2)通过检索增强生成(RAG)获取的外部证据;(3)使用多模型共识策略的聚合知识。我们评估了开源和商业LLMs在三个多样化的现实KGs上的表现。FactCheck还包含一个RAG数据集,包含200万以上的文档,专门用于KG事实验证。此外,我们还提供了一个交互式探索平台,用于分析验证决策。实验分析表明,尽管LLMs显示出有希望的结果,但它们仍然不够稳定和可靠,无法用于现实中的KG验证场景。通过RAG方法整合外部证据会产生波动的性能,提供不一致的改进,相对于更流畅的方法,计算成本更高。同样,基于多模型共识的策略并不总是优于单独的模型,这突显了缺乏万能解决方案的不足。这些发现进一步强调了需要像FactCheck这样的基准来系统地评估和推动这项困难但至关重要的任务。

英文摘要:

Knowledge Graphs (KGs) store structured factual knowledge by linking entities through relationships, crucial for many applications. These applications depend on the KG's factual accuracy, so verifying facts is essential, yet challenging. Expert manual verification is ideal but impractical on a large scale. Automated methods show promise but are not ready for real-world KGs. Large Language Models (LLMs) offer potential with their semantic understanding and knowledge access, yet their suitability and effectiveness for KG fact validation remain largely unexplored. In this paper, we introduce FactCheck, a benchmark designed to evaluate LLMs for KG fact validation across three key dimensions: (1) LLMs internal knowledge; (2) external evidence via Retrieval-Augmented Generation (RAG); and (3) aggregated knowledge employing a multi-model consensus strategy. We evaluated open-source and commercial LLMs on three diverse real-world KGs. FactCheck also includes a RAG dataset with 2+ million documents tailored for KG fact validation. Additionally, we offer an interactive exploration platform for analyzing verification decisions. The experimental analyses demonstrate that while LLMs yield promising results, they are still not sufficiently stable and reliable to be used in real-world KG validation scenarios. Integrating external evidence through RAG methods yields fluctuating performance, providing inconsistent improvements over more streamlined approaches -- at higher computational costs. Similarly, strategies based on multi-model consensus do not consistently outperform individual models, underscoring the lack of a one-fits-all solution. These findings further emphasize the need for a benchmark like FactCheck to systematically evaluate and drive progress on this difficult yet crucial task.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑