RAD-Bench：评估大语言模型在检索增强对话中的能力
RAD-Bench: Evaluating Large Language Models Capabilities in Retrieval Augmented Dialogues
AI总结：
针对现有基准缺乏多轮检索增强对话评估的空白，提出RAD-Bench基准，评估大模型的检索综合与推理能力，实验发现多轮约束增加时模型性能会下降。
摘要：
在大语言模型（LLMs）的实际应用中，常采用外部检索机制——如搜索增强生成（SAG）、工具利用和检索增强生成（RAG）——来提升对话中增强生成内容的质量。这些方法通常涉及多轮对话，每一次交互都会通过从外部来源检索到的相关信息得到丰富。现有的基准要么评估LLMs在多轮对话中的聊天能力，要么评估其在单轮场景下利用检索生成增强回复的能力。然而，在评估LLMs跨多轮利用检索生成更精准回复的能力方面仍存在空白。为解决这一局限，我们提出了RAD-Bench（检索增强对话基准），这一基准旨在评估LLMs在检索后的多轮对话中的能力，这对其在上下文丰富的应用中的部署至关重要。RAD-Bench评估LLMs的两项核心能力：检索综合与检索推理。我们通过判别式问题、检索到的上下文以及对应的参考答案来衡量这两项能力，评估LLMs整合上下文并基于上下文推理，从而在多轮对话中维持并提升对话质量的效果。我们对常用LLMs的评估结果显示，即使提供了准确的检索上下文，随着对话轮次中附加更多层的条件或约束，模型性能会出现下降。相关数据和代码可在https://github.com/mtkresearch/RAD-Bench获取。
英文摘要：
In real-world applications with Large Language Models (LLMs), external retrieval mechanisms - such as Search-Augmented Generation (SAG), tool utilization, and Retrieval-Augmented Generation (RAG) - are often employed to enhance the quality of augmented generations in dialogues. These approaches often come with multi-turn dialogue, where each interaction is enriched by relevant information retrieved from external sources. Existing benchmarks either assess LLMs' chat abilities in multi-turn dialogues or their use of retrieval for augmented responses in single-turn settings. However, there is a gap in evaluating LLMs' ability to leverage retrieval for more precise responses across multiple turns. To address this limitation, we introduce RAD-Bench (Retrieval Augmented Dialogue), a benchmark designed to evaluate LLMs' capabilities in multi-turn dialogues following retrievals, essential for their deployment in context-rich applications. RAD-Bench evaluates two key abilities of LLMs: Retrieval Synthesis and Retrieval Reasoning. These are measured using discriminative questions and retrieved contexts, and corresponding reference answers, assessing how effectively LLMs integrate and reason with context to maintain and enhance conversation quality over multiple turns. Our evaluation results on commonly used LLMs reveal that model performance deteriorates as additional layers of conditions or constraints are applied across conversation turns, even when accurate retrieved contexts are provided. The data and code are available at https://github.com/mtkresearch/RAD-Bench
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
