通过基于检索的蒸馏训练任务专家
Training Task Experts through Retrieval Based Distillation
AI总结：
针对专门任务缺乏高质量数据、LLM 合成数据易重复或错误的问题，提出 ReBase，通过在线检索、领域转换与思维链蒸馏训练任务专家，并在多个基准上取得显著提升。
摘要：
为专门任务创建可部署模型最可靠的方法之一，是获取足量高质量的任务专用数据。然而，对于专门任务而言，这类数据集通常并不存在。现有方法通过从大型语言模型（LLM）生成此类数据，再将这些知识蒸馏到较小模型中来解决该问题。但是，这些方法受限于 LLM 输出的质量，并且往往会生成重复或错误的数据。在本工作中，我们提出 Retrieval Based Distillation（ReBase，基于检索的蒸馏），该方法首先从丰富的在线来源检索数据，然后将其转换为领域专用数据。这一方法显著提升了数据多样性。此外，ReBase 会生成 Chain-of-Thought（思维链）推理，并蒸馏 LLM 的推理能力。我们在 4 个基准测试上评估了该方法，结果表明，本方法显著提升了性能：在 SQuAD 上最高提升 7.8%，在 MNLI 上提升 1.37%，在 BigBench-Hard 上提升 1.94%。
英文摘要：
One of the most reliable ways to create deployable models for specialized tasks is to obtain an adequate amount of high-quality task-specific data. However, for specialized tasks, often such datasets do not exist. Existing methods address this by creating such data from large language models (LLMs) and then distilling such knowledge into smaller models. However, these methods are limited by the quality of the LLMs output, and tend to generate repetitive or incorrect data. In this work, we present Retrieval Based Distillation (ReBase), a method that first retrieves data from rich online sources and then transforms them into domain-specific data. This method greatly enhances data diversity. Moreover, ReBase generates Chain-of-Thought reasoning and distills the reasoning capacity of LLMs. We test our method on 4 benchmarks and results show that our method significantly improves performance by up to 7.8% on SQuAD, 1.37% on MNLI, and 1.94% on BigBench-Hard.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
