发表机构
Texas A&M University（德克萨斯A&M大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究发现多轮对话导致RAG系统性能显著下降，最高相对降幅达21%，不可靠性增加47%，并识别出两种失败模式：翻译扭曲检索查询或LLM未能综合跨轮证据。
AI 中文摘要
在与大型语言模型（LLMs）对话时，用户通常从一个简单问题开始，并通过后续轮次逐步构建出多跳问题。检索增强生成（RAG）及其基于图的变体（GraphRAG）已成为将LLM响应锚定于外部证据的主流方法，然而两者几乎仅在单轮、完全指定的查询上进行评估。我们通过一项大规模模拟研究系统地考察了这一评估错配问题。基于先前关于多轮LLM评估的工作，我们将多跳问答（QA）基准中的问题转化为欠指定的对话，并评估了十个LLM助手搭配八个检索系统在150万次模拟对话中的表现。我们的研究发现，多轮交互导致了广泛的性能退化，造成高达21%的相对性能下降，并使不可靠性增加47%，使得RAG系统同时变得准确度更低、可靠性更差。我们识别出这一退化背后的两种不同失败模式：系统要么“迷失在翻译中”，即对话中的重新表述扭曲了检索查询；要么“迷失在对话中”，即检索成功但LLM未能综合分布在多轮中的证据。
英文摘要
When conversing with large language models (LLMs), users often begin with a simple question and build towards a multi-hop question through follow-up turns. Retrieval-augmented generation (RAG) and its graph-based variant (GraphRAG) have become the dominant approaches for grounding LLM responses in external evidence, yet both are evaluated almost exclusively on single-turn, fully specified queries. We systematically investigate this evaluation mismatch through a large-scale simulation study. Building on prior work on multi-turn LLM evaluation, we transform questions from multi-hop question answering (QA) benchmarks into underspecified conversations and evaluate ten LLM assistants with eight retrieval systems across 1.5 million simulated conversations. Our findings reveal that multi-turn interaction causes widespread performance degradation, incurring relative performance drops of up to 21% and increasing unreliability by 47%, making RAG systems simultaneously less accurate and less reliable. We identify two distinct failure modes behind this degradation. Systems are either lost in translation, where conversational rephrasing distorts the retrieval query, or lost in conversation, where retrieval succeeds but the LLM fails to synthesize evidence distributed across turns.
Comments35 pages, 11 figures
