基于对话记录主动学习增强RAG：拒绝无法回答的问题，回答能够回答的问题
Enhancing RAG with Active Learning on Conversation Records: Reject Incapables and Answer Capables
AI总结：
针对RAG仍难以完全防止LLM幻觉响应且高质量数据集稀缺的问题，本文提出利用对话记录通过主动学习构建数据集，引入AL4RAG及新的样本距离测量方法，训练LLM避开易幻觉问题并准确回答可处理问题，实验表现优于基线。
摘要：
检索增强生成（RAG）是利用外部知识并减少大型语言模型（LLMs）幻觉的关键技术。然而，RAG仍难以完全防止幻觉响应。为解决这一问题，必须识别易产生幻觉的样本或引导LLMs生成正确响应，专家随后对这些样本进行标注以开发高质量数据集来优化LLMs。但此类数据集日益稀缺，导致其创建具有挑战性。本文提出利用LLM广泛使用产生的大量对话来构建这些数据集，训练LLMs避开易产生幻觉的问题，同时准确回答可处理的问题。鉴于专家标注所有对话记录不切实际，本文引入AL4RAG，该方法使用主动学习选择最适合标注的对话样本，在标注预算内优化性能。此外，由于传统主动学习方法的距离度量不适合RAG，我们为RAG主动学习开发了一种新颖的样本距离测量方法。大量实验表明，我们的方法在多个指标上持续优于基线。
英文摘要：
Retrieval-augmented generation (RAG) is a key technique for leveraging external knowledge and reducing hallucinations in large language models (LLMs). However, RAG still struggles to fully prevent hallucinated responses. To address this, it is essential to identify samples prone to hallucination or guide LLMs toward correct responses, which experts then annotate to develop high-quality datasets for refining LLMs. However, the growing scarcity of such datasets makes their creation challenging. This paper proposes using the vast amount of conversations from widespread LLM usage to build these datasets, training LLMs to avoid hallucination-prone questions while accurately responding to manageable ones. Given the impracticality of expert-annotating all conversation records, the paper introduces AL4RAG, which uses active learning to select the most suitable conversation samples for annotation, optimizing performance within an annotation budget. Additionally, recognizing that traditional active learning methods are not fully compatible with RAG due to unsuitable distance metrics, we develop a novel sample distance measurement for RAG active learning. Extensive experiments show that our method consistently outperforms baselines across multiple metrics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
