Vendi-RAG:自适应权衡多样性与质量显著提升大语言模型的检索增强生成
Vendi-RAG: Adaptively Trading-Off Diversity And Quality Significantly Improves Retrieval Augmented Generation With LLMs
AI总结:
Vendi-RAG通过迭代联合优化检索多样性与答案质量,利用Vendi分数和LLM评判器,在多跳问答数据集上显著提升准确性,且优势随文档数增加而增强,并适用于多种LLM。
摘要:
检索增强生成(RAG)通过利用外部知识源,增强了大型语言模型(LLMs)在特定领域问答(QA)任务中的表现。然而,传统的RAG系统主要关注基于相关性的检索,并且常常面临冗余问题,尤其是在推理需要连接多个来源的信息时。本文介绍了Vendi-RAG,一个基于迭代过程的框架,该过程联合优化检索多样性和答案质量。这种联合优化显著提高了多跳问答任务的准确性。Vendi-RAG利用Vendi分数(VS),一种灵活的基于相似性的多样性度量,来促进文档检索中的语义多样性。然后,它使用一个LLM评判器,在推理步骤后评估候选答案,并输出一个分数,检索器利用该分数在每次迭代中平衡检索文档中的相关性和多样性。在三个具有挑战性的数据集——HotpotQA、MuSiQue和2WikiMultiHopQA上的实验证明了Vendi-RAG在多跳推理任务中的有效性。该框架相比传统的单步和多步RAG方法实现了显著的准确性提升,与当前最佳基线Adaptive-RAG相比,在HotpotQA上准确性提升高达+4.2%,在2WikiMultiHopQA上提升+4.1%,在MuSiQue上提升+1.3%。随着检索文档数量的增加,Vendi-RAG的优势更加明显。最后,我们在不同的LLM骨干模型上评估了Vendi-RAG,包括GPT-3.5、GPT-4和GPT-4o-mini,并观察到一致的改进,表明该框架的优势与模型无关。
英文摘要:
Retrieval-augmented generation (RAG) enhances large language models (LLMs) for domain-specific question-answering (QA) tasks by leveraging external knowledge sources. However, traditional RAG systems primarily focus on relevance-based retrieval and often struggle with redundancy, especially when reasoning requires connecting information from multiple sources. This paper introduces Vendi-RAG, a framework based on an iterative process that jointly optimizes retrieval diversity and answer quality. This joint optimization leads to significantly higher accuracy for multi-hop QA tasks. Vendi-RAG leverages the Vendi Score (VS), a flexible similarity-based diversity metric, to promote semantic diversity in document retrieval. It then uses an LLM judge that evaluates candidate answers, generated after a reasoning step, and outputs a score that the retriever uses to balance relevance and diversity among the retrieved documents during each iteration. Experiments on three challenging datasets -- HotpotQA, MuSiQue, and 2WikiMultiHopQA -- demonstrate Vendi-RAG's effectiveness in multi-hop reasoning tasks. The framework achieves significant accuracy improvements over traditional single-step and multi-step RAG approaches, with accuracy increases reaching up to +4.2% on HotpotQA, +4.1% on 2WikiMultiHopQA, and +1.3% on MuSiQue compared to Adaptive-RAG, the current best baseline. The benefits of Vendi-RAG are even more pronounced as the number of retrieved documents increases. Finally, we evaluated Vendi-RAG across different LLM backbones, including GPT-3.5, GPT-4, and GPT-4o-mini, and observed consistent improvements, demonstrating that the framework's advantages are model-agnostic.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。