CTPD: 跨分词器偏好蒸馏
CTPD: Cross Tokenizer Preference Distillation
AI总结：
CTPD提出一种跨分词器的偏好蒸馏框架，通过对齐跨度投影、跨分词器重要性采样和教师锚定参考，实现不同分词器模型间的偏好信息高效转移。
摘要：
尽管知识蒸馏已在预训练和指令微调中得到广泛应用，但将其应用于对齐语言模型与人类偏好仍处于探索阶段，尤其是在更现实的跨分词器设置中。教师模型和学生模型之间的分词方案不兼容性极大地阻碍了细粒度、白盒的偏好信息蒸馏。为解决这一缺口，我们提出了跨分词器偏好蒸馏（CTPD），这是首个统一框架，用于在具有异质分词器的模型之间转移对齐人类行为。CTPD引入了三个关键创新：（1）对齐跨度投影，将教师和学生分词映射到共享的字符级跨度，以实现精确的监督转移；（2）一种跨分词器的分词级重要性采样（TIS-DPO）变体，以改进信用分配；（3）教师锚定参考，允许学生直接利用教师的偏好，在DPO式目标中。我们的理论分析将CTPD建立在重要性采样之上，多个基准实验确认了其有效性，显著优于现有方法。这些结果确立了CTPD作为跨多样分词方案的实用且通用的偏好蒸馏解决方案，为更可访问和高效的语言模型对齐打开了大门。
英文摘要：
While knowledge distillation has seen widespread use in pre-training and instruction tuning, its application to aligning language models with human preferences remains underexplored, particularly in the more realistic cross-tokenizer setting. The incompatibility of tokenization schemes between teacher and student models has largely prevented fine-grained, white-box distillation of preference information. To address this gap, we propose Cross-Tokenizer Preference Distillation (CTPD), the first unified framework for transferring human-aligned behavior between models with heterogeneous tokenizers. CTPD introduces three key innovations: (1) Aligned Span Projection, which maps teacher and student tokens to shared character-level spans for precise supervision transfer; (2) a cross-tokenizer adaptation of Token-level Importance Sampling (TIS-DPO) for improved credit assignment; and (3) a Teacher-Anchored Reference, allowing the student to directly leverage the teacher's preferences in a DPO-style objective. Our theoretical analysis grounds CTPD in importance sampling, and experiments across multiple benchmarks confirm its effectiveness, with significant performance gains over existing methods. These results establish CTPD as a practical and general solution for preference distillation across diverse tokenization schemes, opening the door to more accessible and efficient alignment of language models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
