DomainRAG：一个用于评估领域特定检索增强生成的中文基准
DomainRAG: A Chinese Benchmark for Evaluating Domain-specific Retrieval-Augmented Generation
AI总结：
本文提出中文领域基准DomainRAG，在高校招生场景下从六项能力评估多种主流LLM的RAG表现，证明闭卷模型不足并指出现有RAG的改进方向。
摘要：
检索增强生成（Retrieval-Augmented Generation，RAG）为解决大型语言模型（Large Language Models，LLMs）的多种局限提供了一种有前景的方案，例如幻觉以及难以跟上实时更新等问题。该方法在专家级和领域特定应用中尤为关键，因为LLM难以覆盖专业知识。因此，在此类场景中评估RAG模型至关重要，但现有研究通常依赖维基百科等通用知识来源来评估模型解决常识性问题的能力。本文在高校招生这一领域特定情境下，通过RAG设置评估LLM。我们识别出RAG模型所需的六项能力，包括对话式RAG能力、结构信息分析能力、对外部知识的忠实性、去噪能力、解决时间敏感问题的能力，以及理解多文档交互的能力。每项能力都配有相关数据集，并使用共享语料库来评估RAG模型的表现。我们评估了Llama、Baichuan、ChatGLM和GPT模型等流行LLM。实验结果表明，现有闭卷LLM在处理领域特定问题时存在困难，凸显了使用RAG模型解决专家问题的必要性。此外，RAG模型在理解对话历史、分析结构信息、去噪、处理多文档交互以及忠实于专家知识等方面仍有提升空间。我们期待未来研究能够更好地解决这些问题。
英文摘要：
Retrieval-Augmented Generation (RAG) offers a promising solution to address various limitations of Large Language Models (LLMs), such as hallucination and difficulties in keeping up with real-time updates. This approach is particularly critical in expert and domain-specific applications where LLMs struggle to cover expert knowledge. Therefore, evaluating RAG models in such scenarios is crucial, yet current studies often rely on general knowledge sources like Wikipedia to assess the models' abilities in solving common-sense problems. In this paper, we evaluated LLMs by RAG settings in a domain-specific context, college enrollment. We identified six required abilities for RAG models, including the ability in conversational RAG, analyzing structural information, faithfulness to external knowledge, denoising, solving time-sensitive problems, and understanding multi-document interactions. Each ability has an associated dataset with shared corpora to evaluate the RAG models' performance. We evaluated popular LLMs such as Llama, Baichuan, ChatGLM, and GPT models. Experimental results indicate that existing closed-book LLMs struggle with domain-specific questions, highlighting the need for RAG models to solve expert problems. Moreover, there is room for RAG models to improve their abilities in comprehending conversational history, analyzing structural information, denoising, processing multi-document interactions, and faithfulness in expert knowledge. We expect future studies could solve these problems better.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
