arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.13258cs.CL
收藏

知识选择如何助力检索增强生成?

How Does Knowledge Selection Help Retrieval Augmented Generation?

AWS AI Labs(AWS人工智能实验室);University of Texas at Dallas(德克萨斯大学达拉斯分校)

Xiangci Li, Jessica Ouyang

AI总结:

本文通过可控混合黄金与干扰知识的模拟实验,实证分析知识选择对RAG系统生成性能的影响,发现生成器能力、任务复杂度会决定知识选择的作用程度,为RAG系统优化提供依据。

摘要:

检索增强生成(Retrieval-augmented generation, RAG)是一种通过将外部知识整合到模型输出中以增强自然语言生成的有效方法。尽管已有研究证实了优化知识检索对提升生成质量的重要性,但知识选择(又称重排序或过滤)的作用仍不明确。本文通过实证分析了知识选择如何影响RAG系统的下游生成性能。我们通过将黄金知识与干扰知识进行可控混合,模拟不同的检索与选择条件,以此评估这些因素对生成结果的影响。研究结果表明,下游生成器模型的能力、任务与数据集的复杂度,都会显著影响知识选择对RAG系统整体性能的作用效果。在典型场景下,提升知识召回率是改善生成结果的关键;当使用强生成器模型处理清晰、定义明确的任务时,知识选择器的增益有限。而对于能力较弱的生成器模型,或是更模糊的任务与数据集,知识F1值成为关键因素,知识选择器在提升整体性能方面会发挥更突出的作用。

英文摘要:

Retrieval-augmented generation (RAG) is a powerful method for enhancing natural language generation by integrating external knowledge into a model's output. While prior work has demonstrated the importance of improving knowledge retrieval for boosting generation quality, the role of knowledge selection, a.k.a. reranking or filtering, remains less clear. This paper empirically analyzes how knowledge selection influences downstream generation performance in RAG systems. By simulating different retrieval and selection conditions through a controlled mixture of gold and distractor knowledge, we assess the impact of these factors on generation outcomes. Our findings indicate that the downstream generator model's capability, as well as the complexity of the task and dataset, significantly influence the impact of knowledge selection on the overall RAG system performance. In typical scenarios, improving the knowledge recall score is key to enhancing generation outcomes, with the knowledge selector providing limited benefit when a strong generator model is used on clear, well-defined tasks. For weaker generator models or more ambiguous tasks and datasets, the knowledge F1 score becomes a critical factor, and the knowledge selector plays a more prominent role in improving overall performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑