arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.05239cs.CLcs.AI
收藏

增强知识图谱构建:以幻觉、遗漏和图相似度指标为重点的评估

Enhancing Knowledge Graph Construction: Evaluating with Emphasis on Hallucination, Omission, and Graph Similarity Metrics

CNRS(法国国家科学研究中心);Université de Bourgogne(勃艮第大学);ICB(布尔戈尼计算机科学实验室)

Hussam Ghanem, Christophe Cruz

AI总结:

本文针对知识图谱构建中的幻觉与遗漏问题,提出引入BERTScore和95%图匹配阈值的增强评估框架,通过Mistral模型微调实验证明其能提升构建准确性但泛化能力下降,强调全面评估指标的重要性。

摘要:

大语言模型的最新进展展示了从非结构化文本自动构建知识图谱的巨大潜力。本文基于我们之前的工作[16]——该工作使用精确率、召回率、F1分数、三元组匹配和图匹配等指标评估了多种模型——提出了一种改进方法,以解决幻觉和遗漏这两个关键问题。我们提出了一个增强的评估框架,引入BERTScore用于图相似度评估,并为图匹配设定了95%的实用阈值。我们的实验聚焦于Mistral模型,在零样本和少样本设置下比较其原始版本和微调版本。我们进一步使用KELM-sub训练数据集中的示例扩展实验,结果表明微调后的模型显著提高了知识图谱构建的准确性,同时减少了精确的幻觉和遗漏。然而,我们的发现也揭示出微调后的模型在KELM-sub数据集上的泛化任务中表现较差。本研究强调了全面评估指标在推动从文本数据构建知识图谱的前沿技术水平方面的重要性。

英文摘要:

Recent advancements in large language models have demonstrated significant potential in the automated construction of knowledge graphs from unstructured text. This paper builds upon our previous work [16], which evaluated various models using metrics like precision, recall, F1 score, triple matching, and graph matching, and introduces a refined approach to address the critical issues of hallucination and omission. We propose an enhanced evaluation framework incorporating BERTScore for graph similarity, setting a practical threshold of 95% for graph matching. Our experiments focus on the Mistral model, comparing its original and fine-tuned versions in zero-shot and few-shot settings. We further extend our experiments using examples from the KELM-sub training dataset, illustrating that the fine-tuned model significantly improves knowledge graph construction accuracy while reducing the exact hallucination and omission. However, our findings also reveal that the fine-tuned models perform worse in generalization tasks on the KELM-sub dataset. This study underscores the importance of comprehensive evaluation metrics in advancing the state-of-the-art in knowledge graph construction from textual data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑