无偏Top-$k$估计用于同策略蒸馏
Unbiased Top-$k$ Estimation for On-Policy Distillation
- Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
- State Key Laboratory of Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）
机构由 AI 辅助整理，请以论文原文为准。
中文总结 AI 辅助
针对同策略蒸馏中Top-k估计的偏差问题，提出尾校正Top-k同策略蒸馏（TT-OPD），结合top-k token与采样token实现无偏梯度估计，显著提升性能。
中文摘要 AI 辅助
同策略蒸馏（OPD）正成为大语言模型（LLM）后训练的重要组成部分，用于将强教师LLM的推理能力迁移到较弱的学生LLM。OPD通过最小化教师与学生之间的反向KL散度来训练学生，其中使用学生策略生成的轨迹（rollouts）。然而，在OPD中估计反向KL散度的梯度仍然是一个挑战。仅使用学生生成轨迹中的采样token在计算上廉价，但提供的分布监督有限，这会降低准确性。此外，使用完整词汇表提供完整的分布监督，但计算成本高昂。因此，近期工作提出了Top-$k$ OPD（TK-OPD），使用选定的top-$k$个token，相比采样token估计提供更丰富的分布监督，且计算成本远低于全词汇表估计。不幸的是，仅使用选定的top-$k$个token会引入偏差，导致准确性下降，因为选定top-$k$个token之外的概率质量被丢弃。为了解决TK-OPD的偏差问题，我们提出了尾校正Top-$k$同策略蒸馏（TT-OPD）。它保留了TK-OPD的优势，包括丰富的分布监督和低计算成本，同时提供了反向KL散度梯度的无偏估计器。TT-OPD的关键见解是不仅使用选定的top-$k$个token，还使用学生生成轨迹中的采样token，从而在期望上恢复被丢弃的概率质量，避免偏差。实验结果表明，TT-OPD显著优于其他测试的OPD变体。
英文摘要
On-policy distillation (OPD) is becoming an important component of large language model (LLM) post-training for transferring the reasoning capability of a strong teacher LLM to a weaker student LLM. OPD trains the student by minimizing the reverse KL divergence between the teacher and the student via rollouts generated by the student's policy. However, estimating the gradient of the reverse KL divergence in OPD remains a challenge. Using only the sampled token from the student-generated rollout is computationally cheap but provides limited distributional supervision, which will degrade accuracy. In addition, using the full vocabulary provides complete distributional supervision but is computationally expensive. Therefore, recent works propose Top-$k$ OPD (TK-OPD) that use selected top-$k$ tokens, which provides richer distributional supervision than sampled-token estimation at substantially lower computational cost than full-vocabulary estimation. Unfortunately, using only the selected top-$k$ tokens induces bias, leading to accuracy degradation, as the probability mass outside the selected top-$k$ tokens is discarded. To address the bias of TK-OPD, we propose Tail-Corrected Top-$k$ On-Policy Distillation (TT-OPD). It preserves the advantages of TK-OPD, including rich distributional supervision and low computational cost, while providing an unbiased estimator of the gradient of the reverse KL divergence. The key insight of TT-OPD is to use not only the selected top-$k$ tokens, but also the sampled token from the student-generated rollout, thereby recovering the discarded probability mass in expectation, avoiding the bias. Experimental results demonstrate that TT-OPD significantly outperforms other tested OPD variants.
