arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.19601cs.CL
收藏

PA1-OPD:用于语言模型预训练的单步前缀增广在线策略蒸馏

PA1-OPD: One-Step Prefix-Augmented On-Policy Distillation for Language Model Pretraining

Huawei Technologies(华为技术有限公司);Beihang University(北京航空航天大学)

Zhenhe Wu, Jian Yang, Yaping Jin, Xinlai Kang, Jie Hu, Hanting Chen

AI总结:

本研究提出PA1-OPD方法,通过增广语料库路径蒸馏扩展监督状态,在持续预训练的多个数据集上提升了模型的保留交叉熵,验证了学生诱导局部监督对预训练蒸馏的补充价值。

摘要:

知识蒸馏(KD)通过在语料库前缀匹配教师分布来改进语言模型预训练,但在学生预测诱导的上下文处未提供直接监督。我们提出PA1-OPD,它在每个选定前缀的一次学生采样转换后,用教师反馈增广语料库路径蒸馏,在保留下一个词元预测和语料库路径KD的同时扩展了监督状态。数据锚定根支持并行采样,可微树注意力共享前缀计算且不改变推理。在两个持续预训练设置中,全覆盖在所有评估检查点上,相对于KD在C4、PG-19和Pile上的保留交叉熵均有提升;在第二阶段,选择学生意外值最高的10%根,使KD相对于下一个词元训练的宏观交叉熵增益增加48.85%。当分支监督排除已在语料库路径上的后继状态时,增益仍然存在,支持超出观测上下文的反馈价值。额外的深度或广度在每一项监督权重不变时改善了聚合结果,收益因语料库而异。这些发现支持将学生诱导的局部监督作为预训练蒸馏的补充。

英文摘要:

Knowledge distillation (KD) improves language-model pretraining by matching teacher distributions at corpus prefixes, but provides no direct supervision at contexts induced by student predictions. We introduce PA1-OPD, which augments corpus-path distillation with teacher feedback after one student-sampled transition per selected prefix. This expands the supervised states while retaining next-token prediction and corpus-path KD. Data-anchored roots support parallel sampling, and differentiable tree attention shares prefix computation without changing inference. In two continued-pretraining settings, full coverage improves held-out cross-entropy over KD across C4, PG-19, and Pile at every evaluated checkpoint. In Stage2, selecting the 10% of roots with highest student surprisal adds 48.85% to KD's Macro cross-entropy gain over next-token training. Gains remain when branch supervision excludes successor states already on the corpus path, supporting the value of feedback beyond observed contexts. Additional depth or breadth improves aggregate results at unchanged per-term supervision weight, with benefits varying by corpus. These findings support local student-induced supervision as a complement to pretraining distillation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑