中间蒸馏：从黑盒大语言模型进行数据高效的信息检索蒸馏
Intermediate Distillation: Data-Efficient Distillation from Black-Box LLMs for Information Retrieval
AI总结：
针对黑盒 LLM 无法通过权重或输出概率提供蒸馏监督的问题，提出仅用排序生成结果的 LLM-ranker-retriever 中间蒸馏方案，以 1,000 个实例提升检索器及 RAG 问答性能。
摘要：
近期研究探索了从大语言模型（LLM）中蒸馏知识，以优化检索器模型，尤其是在检索增强生成（RAG）框架内。然而，现有大多数训练方法依赖于从 LLM 的权重或其输出概率中提取监督信号，这不仅资源消耗大，而且与黑盒 LLM 不兼容。本文提出 Intermediate Distillation（中间蒸馏），这是一种数据高效的知识蒸馏训练方案，将 LLM 视为黑盒，并通过创新的 LLM-ranker-retriever（大语言模型-排序器-检索器）流水线蒸馏其知识，仅使用 LLM 的排序生成结果作为监督信号。大量实验表明，所提方法仅需 1,000 个训练实例即可显著提升检索器模型的性能。此外，经蒸馏的检索器模型显著提升了 RAG 框架内问答任务的性能，证明了 LLM 经济且有效地训练较小模型的潜力。
英文摘要：
Recent research has explored distilling knowledge from large language models (LLMs) to optimize retriever models, especially within the retrieval-augmented generation (RAG) framework. However, most existing training methods rely on extracting supervision signals from LLMs' weights or their output probabilities, which is not only resource-intensive but also incompatible with black-box LLMs. In this paper, we introduce \textit{Intermediate Distillation}, a data-efficient knowledge distillation training scheme that treats LLMs as black boxes and distills their knowledge via an innovative LLM-ranker-retriever pipeline, solely using LLMs' ranking generation as the supervision signal. Extensive experiments demonstrate that our proposed method can significantly improve the performance of retriever models with only 1,000 training instances. Moreover, our distilled retriever model significantly boosts performance in question-answering tasks within the RAG framework, demonstrating the potential of LLMs to economically and effectively train smaller models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
