助教在环：在低预算场景下改进来自不完美教师模型的知识蒸馏
Teaching-Assistant-in-the-Loop: Improving Knowledge Distillation from Imperfect Teacher Models in Low-Budget Scenarios
AI总结：
本文提出一种三组件框架，利用学生自一致性信号和助教模型置信度评分信号，在低预算且教师模型不完美的场景下改进知识蒸馏，并通过两阶段训练在四个复杂推理任务上实现最高20.79%的相对提升。
摘要：
人们越来越关注将大语言模型（LLM）中的任务特定知识蒸馏到较小的学生模型中。然而，LLM蒸馏面临双重挑战：1）查询教师LLM（如GPT-4）以收集足够数量的演示样本会带来高昂成本；2）教师LLM可能提供不完美的输出，对学生模型的学习过程产生负面影响。为了在资源受限且教师不完美的场景下提高样本效率，我们提出了一个利用三种信号类型的三组件框架。第一种信号是学生模型的自一致性（学生多次输出的一致性），它是学生模型置信度的代理指标。具体来说，我们引入一个“助教”（TA）模型，通过置信度评分来评估学生模型和教师模型输出的不确定性，这构成用于学生训练的另外两种信号。此外，我们提出了一种两阶段训练方案，首先用少量数据对学生模型进行预热，以更好地利用学生模型的信号。实验表明，我们提出的框架在四个复杂推理任务上具有优越性。平均而言，与不使用任何信号的微调相比，我们提出的两阶段框架在跨数据集上带来了最高达20.79%的相对改进。
英文摘要：
There is increasing interest in distilling task-specific knowledge from large language models (LLM) to smaller student models. Nonetheless, LLM distillation presents a dual challenge: 1) there is a high cost associated with querying the teacher LLM, such as GPT-4, for gathering an ample number of demonstrations; 2) the teacher LLM might provide imperfect outputs with a negative impact on the student's learning process. To enhance sample efficiency within resource-constrained, imperfect teacher scenarios, we propose a three-component framework leveraging three signal types. The first signal is the student's self-consistency (consistency of student multiple outputs), which is a proxy of the student's confidence. Specifically, we introduce a ``teaching assistant'' (TA) model to assess the uncertainty of both the student's and the teacher's outputs via confidence scoring, which serves as another two signals for student training. Furthermore, we propose a two-stage training schema to first warm up the student with a small proportion of data to better utilize student's signal. Experiments have shown the superiority of our proposed framework for four complex reasoning tasks. On average, our proposed two-stage framework brings a relative improvement of up to 20.79% compared to fine-tuning without any signals across datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
