实际性能提升有多显著?GraphRAG的无偏评估框架
How Significant Are the Real Performance Gains? An Unbiased Evaluation Framework for GraphRAG
AI总结:
针对GraphRAG现有评估框架的不相关问题与评估偏差缺陷,提出无偏评估框架并发现代表性GraphRAG方法的性能提升远小于此前报告结果,呼吁开展科学评估。
摘要:
基于知识图谱检索上下文的图检索增强生成(GraphRAG)可增强大语言模型(LLM)为用户问题生成高质量答案,目前已提出的多种GraphRAG方法均报告了令人鼓舞的答案质量性能。但我们发现当前GraphRAG的答案评估框架存在两个关键缺陷:不相关问题与评估偏差,这可能导致性能结论有偏甚至错误。为解决这两个缺陷,我们提出一种无偏评估框架,该框架使用基于图-文本的问题生成,生成与底层数据集更相关的问题,并采用无偏评估流程消除基于LLM的答案评估中的偏差。我们将该无偏框架应用于评估3种代表性GraphRAG方法,发现它们的性能提升远小于之前报告的结果。尽管我们的评估框架可能仍存在缺陷,但它呼吁开展科学评估,为GraphRAG研究奠定坚实基础。
英文摘要:
By retrieving contexts from knowledge graphs, graph-based retrieval-augmented generation (GraphRAG) enhances large language models (LLMs) to generate quality answers for user questions. Many GraphRAG methods have been proposed and reported inspiring performance in answer quality. However, we observe that the current answer evaluation framework for GraphRAG has two critical flaws, i.e., unrelated questions and evaluation biases, which may lead to biased or even wrong conclusions on performance. To tackle the two flaws, we propose an unbiased evaluation framework that uses graph-text-grounded question generation to produce questions that are more related to the underlying dataset and an unbiased evaluation procedure to eliminate the biases in LLM-based answer assessment. We apply our unbiased framework to evaluate 3 representative GraphRAG methods and find that their performance gains are much more moderate than reported previously. Although our evaluation framework may still have flaws, it calls for scientific evaluations to lay solid foundations for GraphRAG research.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。