充分上下文：检索增强生成系统的新视角
Sufficient Context: A New Lens on Retrieval Augmented Generation Systems
AI总结：
本文提出“充分上下文”概念，用于分析RAG系统中错误来源，发现大模型在上下文不充分时倾向错误而非弃权，并据此提出利用充分上下文信息的选择性生成方法，将正确答案比例提升2-10%。
摘要：
为大型语言模型（LLM）增加上下文可在许多应用中提升性能。尽管关于检索增强生成（RAG）系统的研究众多，但一个悬而未决的问题是：错误究竟源于LLM未能利用检索到的上下文，还是上下文本身不足以回答查询。为阐明这一问题，我们提出了“充分上下文”的新概念，以及一种对具有足够信息回答查询的实例进行分类的方法。随后，我们利用充分上下文分析了多个模型和数据集。通过基于上下文充分性对错误进行分层，我们发现基线性能较高的大型模型（Gemini 1.5 Pro、GPT 4o、Claude 3.5）在上下文充分时擅长回答问题，但在上下文不充分时往往输出错误答案而非弃权（不执行）。另一方面，基线性能较低的小型模型（Mistral 3、Gemma 2）即使在上下文充分的情况下也经常产生幻觉或弃权（不执行）。我们进一步分类了上下文有用且能提高准确性的情况，尽管上下文并未完全回答查询，且没有上下文时模型会出错。基于我们的发现，我们探索了减少RAG系统中幻觉的方法，包括一种利用充分上下文信息进行引导性弃权（不执行）的新型选择性生成方法。我们的方法将Gemini、GPT和Gemma模型在模型响应时正确答案的比例提高了2-10%。关键发现及我们自动评分分析中使用的提示词已发布在我们的GitHub上。
英文摘要：
Augmenting LLMs with context leads to improved performance across many applications. Despite much research on Retrieval Augmented Generation (RAG) systems, an open question is whether errors arise because LLMs fail to utilize the context from retrieval or the context itself is insufficient to answer the query. To shed light on this, we develop a new notion of sufficient context, along with a method to classify instances that have enough information to answer the query. We then use sufficient context to analyze several models and datasets. By stratifying errors based on context sufficiency, we find that larger models with higher baseline performance (Gemini 1.5 Pro, GPT 4o, Claude 3.5) excel at answering queries when the context is sufficient, but often output incorrect answers instead of abstaining when the context is not. On the other hand, smaller models with lower baseline performance (Mistral 3, Gemma 2) hallucinate or abstain often, even with sufficient context. We further categorize cases when the context is useful, and improves accuracy, even though it does not fully answer the query and the model errs without the context. Building on our findings, we explore ways to reduce hallucinations in RAG systems, including a new selective generation method that leverages sufficient context information for guided abstention. Our method improves the fraction of correct answers among times where the model responds by 2--10\% for Gemini, GPT, and Gemma. Key findings and the prompts used in our autorater analysis are available on our github.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
