arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.03181cs.AIcs.CLcs.IR
收藏

C-RAG：检索增强语言模型的认证生成风险

C-RAG: Certified Generation Risks for Retrieval-Augmented Language Models

University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Delft University of Technology（代尔夫特理工大学）；Netflix Eyeline Studios；University of California, Berkeley（加州大学伯克利分校）；University of Chicago（芝加哥大学）

Mintong Kang, Nezihe Merve Gürel, Ning Yu, Dawn Song, Bo Li

备注：
Accepted to ICML 2024
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出C-RAG框架，首次为检索增强语言模型（RAG）提供保形生成风险的理论认证与上置信界保证，并证明在检索模型与transformer质量非平凡时RAG能降低生成风险。

摘要：

尽管大型语言模型（LLMs）在多样化应用中展现出令人印象深刻的能力，但它们仍然面临可信度问题，如幻觉和不对齐。检索增强语言模型（RAG）被提出通过基于外部知识来增强生成内容的可信度，但对它们生成风险的理论理解仍未被探索。在本文中，我们回答：1）RAG是否确实能导致低生成风险；2）如何为RAG和原始LLMs的生成风险提供可证明的保证；3）什么充分条件能使RAG模型降低生成风险。我们提出了C-RAG，这是第一个为RAG模型认证生成风险的框架。具体来说，我们为RAG模型提供了保形风险分析，并认证了生成风险的上置信界，我们将其称为保形生成风险。我们还为测试分布偏移下的一般有界风险函数提供了保形生成风险的理论保证。我们证明，当检索模型和transformer的质量非平凡时，RAG实现了比单个LLM更低的保形生成风险。我们密集的实证结果证明了我们的保形生成风险保证在四个广泛使用的NLP数据集和四个最先进的检索模型上的合理性和紧密性。

英文摘要：

Despite the impressive capabilities of large language models (LLMs) across diverse applications, they still suffer from trustworthiness issues, such as hallucinations and misalignments. Retrieval-augmented language models (RAG) have been proposed to enhance the credibility of generations by grounding external knowledge, but the theoretical understandings of their generation risks remains unexplored. In this paper, we answer: 1) whether RAG can indeed lead to low generation risks, 2) how to provide provable guarantees on the generation risks of RAG and vanilla LLMs, and 3) what sufficient conditions enable RAG models to reduce generation risks. We propose C-RAG, the first framework to certify generation risks for RAG models. Specifically, we provide conformal risk analysis for RAG models and certify an upper confidence bound of generation risks, which we refer to as conformal generation risk. We also provide theoretical guarantees on conformal generation risks for general bounded risk functions under test distribution shifts. We prove that RAG achieves a lower conformal generation risk than that of a single LLM when the quality of the retrieval model and transformer is non-trivial. Our intensive empirical results demonstrate the soundness and tightness of our conformal generation risk guarantees across four widely-used NLP datasets on four state-of-the-art retrieval models.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑