arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.02549cs.CLcs.AI
收藏

基于知识图谱的RAG系统评估框架

Knowledge-Graph Based RAG System Evaluation Framework

Technical University of Munich, Robotics, Artificial Intelligence and Embedded Systems(慕尼黑技术大学,机器人,人工智能与嵌入式系统)

Sicheng Dong, Vahid Zolfaghari, Nenad Petrovic, Alois Knoll

AI总结:

针对传统指标难以有效评估RAG系统的问题,本研究在RAGAS框架基础上扩展出基于知识图谱的评估范式,引入多跳推理等实现更全面的评分,经对比与人工标注验证其对语义差异更敏感。

摘要:

大语言模型(LLMs)已成为重要研究热点,被应用于文本生成、对话系统等诸多领域。LLM最核心的应用之一是检索增强生成(RAG),它能大幅提升生成内容的可靠性与相关性。但RAG系统的评估仍是一项具有挑战性的任务:传统评估指标难以有效捕捉现代LLM生成内容的关键特征,这类内容通常具备高流畅度与自然度。受知名RAG评估框架RAGAS工具的启发,我们将该框架扩展为基于知识图谱(KG)的评估范式,引入多跳推理与语义社区聚类以推导更全面的评分指标。通过纳入这些综合评估标准,我们能更深入地理解RAG系统,对其性能形成更细致的认知。为验证我们方法的有效性,我们将其性能与RAGAS得分进行对比,并构建了一个人工标注子集,以评估人类判断与自动化指标之间的相关性。此外,我们开展了针对性实验,证明我们的基于KG的评估方法对生成输出中的细微语义差异更敏感。最后,我们讨论了RAG系统评估中的关键挑战,并指明了未来研究的潜在方向。

英文摘要:

Large language models (LLMs) has become a significant research focus and is utilized in various fields, such as text generation and dialog systems. One of the most essential applications of LLM is Retrieval Augmented Generation (RAG), which greatly enhances generated content's reliability and relevance. However, evaluating RAG systems remains a challenging task. Traditional evaluation metrics struggle to effectively capture the key features of modern LLM-generated content that often exhibits high fluency and naturalness. Inspired by the RAGAS tool, a well-known RAG evaluation framework, we extended this framework into a KG-based evaluation paradigm, enabling multi-hop reasoning and semantic community clustering to derive more comprehensive scoring metrics. By incorporating these comprehensive evaluation criteria, we gain a deeper understanding of RAG systems and a more nuanced perspective on their performance. To validate the effectiveness of our approach, we compare its performance with RAGAS scores and construct a human-annotated subset to assess the correlation between human judgments and automated metrics. In addition, we conduct targeted experiments to demonstrate that our KG-based evaluation method is more sensitive to subtle semantic differences in generated outputs. Finally, we discuss the key challenges in evaluating RAG systems and highlight potential directions for future research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑