CRUD-RAG：面向大语言模型检索增强生成的综合性中文基准
CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models
AI总结：
针对现有RAG评估基准场景单一、仅评估大模型组件的问题，本文提出中文基准CRUD-RAG，覆盖创建、读取、更新、删除四类场景，全面评估RAG各组件影响并给出优化建议。
摘要：
检索增强生成（Retrieval-Augmented Generation, RAG）是一种通过引入外部知识源增强大语言模型（large language models, LLMs）能力的技术，可解决大语言模型普遍存在的信息过时、易生成不准确“幻觉”内容等局限。然而，RAG系统的评估颇具挑战，现有基准的覆盖范围与多样性十分有限：当前多数基准主要评估问答类应用，忽略了RAG可发挥优势的更广泛场景；且实验中仅评估RAG流水线里大语言模型组件的性能，忽视了检索组件与外部知识库的影响。针对这些问题，本文构建了一个规模更大、更全面的基准，在多种RAG应用场景下对RAG系统的所有组件展开评估。具体而言，我们将RAG应用范围划分为创建（Create）、读取（Read）、更新（Update）、删除（Delete）四类，即CRUD，每类对应一种独特用例：“创建”指需要生成原创、多样化内容的场景；“读取”涉及在知识密集型场景中回答复杂问题；“更新”聚焦于修正已有文本中的错误或不一致之处；“删除”对应将长文本总结为更简洁形式的任务。我们为每类CRUD场景都构建了综合数据集以评估RAG系统性能，还分析了检索器、上下文长度、知识库构建、大语言模型等RAG系统各组件的影响，最终为不同场景下优化RAG技术提供了实用洞见。
英文摘要：
Retrieval-Augmented Generation (RAG) is a technique that enhances the capabilities of large language models (LLMs) by incorporating external knowledge sources. This method addresses common LLM limitations, including outdated information and the tendency to produce inaccurate "hallucinated" content. However, the evaluation of RAG systems is challenging, as existing benchmarks are limited in scope and diversity. Most of the current benchmarks predominantly assess question-answering applications, overlooking the broader spectrum of situations where RAG could prove advantageous. Moreover, they only evaluate the performance of the LLM component of the RAG pipeline in the experiments, and neglect the influence of the retrieval component and the external knowledge database. To address these issues, this paper constructs a large-scale and more comprehensive benchmark, and evaluates all the components of RAG systems in various RAG application scenarios. Specifically, we have categorized the range of RAG applications into four distinct types-Create, Read, Update, and Delete (CRUD), each representing a unique use case. "Create" refers to scenarios requiring the generation of original, varied content. "Read" involves responding to intricate questions in knowledge-intensive situations. "Update" focuses on revising and rectifying inaccuracies or inconsistencies in pre-existing texts. "Delete" pertains to the task of summarizing extensive texts into more concise forms. For each of these CRUD categories, we have developed comprehensive datasets to evaluate the performance of RAG systems. We also analyze the effects of various components of the RAG system, such as the retriever, the context length, the knowledge base construction, and the LLM. Finally, we provide useful insights for optimizing the RAG technology for different scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
