arXiv 2309.014312023-12-20cs.CL

检索增强生成中的大型语言模型基准测试

Benchmarking Large Language Models in Retrieval-Augmented Generation

Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）

Jiawei Chen, Hongyu Lin, Xianpei Han, Le Sun

备注：: Accepted to AAAI 2024

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文系统评估了检索增强生成（RAG）对大型语言模型的影响，构建了中英文基准RGB，测试了6个模型在噪声鲁棒性、负面拒绝、信息整合和反事实鲁棒性四方面能力，发现模型在负面拒绝、信息整合和虚假信息处理上仍有显著困难。

摘要：

检索增强生成（RAG）是缓解大型语言模型（LLMs）幻觉问题的一种有前景的方法。然而，现有研究缺乏对检索增强生成对不同大型语言模型影响的严格评估，这使得识别不同LLMs在RAG能力方面的潜在瓶颈变得具有挑战性。在本文中，我们系统地研究了检索增强生成对大型语言模型的影响。我们分析了不同大型语言模型在RAG所需的4项基本能力上的表现，包括噪声鲁棒性、负面拒绝、信息整合和反事实鲁棒性。为此，我们建立了检索增强生成基准（RGB），这是一个用于RAG评估的新语料库，涵盖英文和中文。RGB根据解决案例所需的上述基本能力，将基准中的实例划分为4个独立的测试平台。然后，我们在RGB上评估了6个具有代表性的LLMs，以诊断当前LLMs在应用RAG时面临的挑战。评估结果显示，虽然LLMs表现出一定程度的噪声鲁棒性，但在负面拒绝、信息整合和处理虚假信息方面仍然存在显著困难。上述评估结果表明，要将RAG有效应用于LLMs，仍有相当长的路要走。

英文摘要：

Retrieval-Augmented Generation (RAG) is a promising approach for mitigating the hallucination of large language models (LLMs). However, existing research lacks rigorous evaluation of the impact of retrieval-augmented generation on different large language models, which make it challenging to identify the potential bottlenecks in the capabilities of RAG for different LLMs. In this paper, we systematically investigate the impact of Retrieval-Augmented Generation on large language models. We analyze the performance of different large language models in 4 fundamental abilities required for RAG, including noise robustness, negative rejection, information integration, and counterfactual robustness. To this end, we establish Retrieval-Augmented Generation Benchmark (RGB), a new corpus for RAG evaluation in both English and Chinese. RGB divides the instances within the benchmark into 4 separate testbeds based on the aforementioned fundamental abilities required to resolve the case. Then we evaluate 6 representative LLMs on RGB to diagnose the challenges of current LLMs when applying RAG. Evaluation reveals that while LLMs exhibit a certain degree of noise robustness, they still struggle significantly in terms of negative rejection, information integration, and dealing with false information. The aforementioned assessment outcomes indicate that there is still a considerable journey ahead to effectively apply RAG to LLMs.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。