学生模型作为噪声教师模型的固有去噪器
Student as an Inherent Denoiser of Noisy Teacher
AI总结：
针对噪声教师模型伪标签影响知识蒸馏性能的问题，研究发现学生模型固有去噪能力，据此提出Peer-Advised KD方法，低数据量下性能超越大模型，可匹敌多数据量的监督微调。
摘要：
知识蒸馏（KD）已被广泛用于在低数据量场景下通过伪标签学习将大语言模型（LLM）的知识迁移到专用模型中。然而，教师模型生成的伪标签通常存在噪声，可能影响KD性能。本研究深入探究了带噪声教师的KD问题，发现学生模型在KD过程中生成的预测结果已经比用于训练它的教师标签更准确，表明其具备对噪声教师标签进行去噪的固有能力。受这一发现启发，我们提出Peer-Advised KD（同伴建议知识蒸馏），以改进来自噪声教师的 vanilla KD（原始知识蒸馏）。实验表明，在仅有50条人工标注数据的情况下，Peer-Advised KD的性能比LLM高出约5%，甚至能与使用750条人工标注数据的标准监督微调效果相当。
英文摘要：
Knowledge distillation (KD) has been widely employed to transfer knowledge from a large language model (LLM) to a specialized model in low-data regimes through pseudo label learning. However, pseudo labels generated by teacher models are usually noisy and may influence KD performance. This study delves into KD with noisy teachers and uncovers that the student model can already generate more accurate predictions than the teacher labels used to train it during KD, indicating its inherent ability to denoise noisy teacher labels. Motivated by this finding, we propose Peer-Advised KD to improve vanilla KD from noisy teachers. Experiments show that Peer-Advised KD can outperform LLM by approximately 5% with 50 human-labeled data, and even competitive to standard supervised finetuning with 750 human-labeled data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
