基于伪目标训练的自然语言生成知识蒸馏系统研究
A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Training
AI总结：
本文系统研究NLG任务特定知识蒸馏，提出伪目标增强与Joint-Teaching方法，并在无标注数据下用GPT-4验证了高压缩率的有效性。
摘要：
现代自然语言生成（NLG）模型具有庞大的计算和存储需求。在本工作中，我们研究压缩这些模型的潜力，这对于服务数百万用户的现实应用至关重要。我们聚焦于知识蒸馏（Knowledge Distillation, KD）技术，其中一个小型学生模型学习模仿一个大型教师模型，从而将知识从教师迁移到学生。与大量先前工作不同，我们的目标是针对特定NLG任务和特定数据集优化模型。通常在现实应用中，除了有标签数据外，还存在大量无标签的任务特定数据，这对于通过KD实现高压缩率至关重要。在本工作中，我们在现实假设下对各种NLG任务的任务特定KD技术进行了系统研究。我们讨论了NLG蒸馏的特殊特征，尤其是曝光偏差问题。随后，我们推导出一族伪目标（Pseudo-Target, PT）增强方法，大幅扩展了先前关于序列级KD的工作。我们提出Joint-Teaching方法，该方法对由教师和学生共同生成的多个PT应用词级KD。最后，我们在一个没有标注样本的极端设置中，使用GPT-4作为教师来验证我们的发现。我们的研究提供了实用的模型设计观察，并证明了PT训练在NLG任务特定KD中的有效性。
英文摘要：
Modern Natural Language Generation (NLG) models come with massive computational and storage requirements. In this work, we study the potential of compressing them, which is crucial for real-world applications serving millions of users. We focus on Knowledge Distillation (KD) techniques, in which a small student model learns to imitate a large teacher model, allowing to transfer knowledge from the teacher to the student. In contrast to much of the previous work, our goal is to optimize the model for a specific NLG task and a specific dataset. Typically in real-world applications, in addition to labeled data there is abundant unlabeled task-specific data, which is crucial for attaining high compression rates via KD. In this work, we conduct a systematic study of task-specific KD techniques for various NLG tasks under realistic assumptions. We discuss the special characteristics of NLG distillation and particularly the exposure bias problem. Following, we derive a family of Pseudo-Target (PT) augmentation methods, substantially extending prior work on sequence-level KD. We propose the Joint-Teaching method, which applies word-level KD to multiple PTs generated by both the teacher and the student. Finally, we validate our findings in an extreme setup with no labeled examples using GPT-4 as the teacher. Our study provides practical model design observations and demonstrates the effectiveness of PT training for task-specific KD in NLG.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
