通过对数似然差异设计领域混合以对齐语言模型与目标模型
Domain Mixture Design via Log-Likelihood Differences for Aligning Language Models with a Target Model
AI总结：
本文通过设计训练数据的领域混合，利用对数似然空间对齐模型方向，减少KL散度并提升下游任务性能。
摘要：
本文研究了通过设计训练数据的领域混合来对齐基础模型与目标模型的方法。通过将模型视为对数似然空间中的点，并将训练更新方向对齐于目标模型方向，确定领域权重。实验表明，与均匀加权相比，该方法能更有效地减少KL散度。尽管知识蒸馏在可用时更有效，但本文方法仍能实现有意义的对齐，且下游任务性能也更接近目标模型的性能。
英文摘要：
Instead of directly distilling a language model, this study addresses the problem of aligning a base model with a target model in distribution by designing the domain mixture of training data for pretraining or continued pretraining as a fixed training recipe. We propose a method for determining domain weights by viewing models as points in log-likelihood space and aligning the training update direction with the direction toward the target model. Experiments with NanoGPT show that the proposed method consistently reduces the KL divergence between the trained base model and the target model relative to training with Pile-original weighting. Although knowledge distillation remains more effective when available, the proposed method achieves meaningful alignment, and downstream task performance also tends to become closer to that of the target model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
