daDPO：用于蒸馏对话能力的分布感知DPO
daDPO: Distribution-Aware DPO for Distilling Conversational Abilities
AI总结：
本文提出daDPO，一种结合输出分布与偏好优化的统一蒸馏方法，通过利用教师模型的分布信息，显著提升剪枝及小规模LLM的对话能力，实验表明其优于现有方法。
摘要：
大型语言模型（LLMs）在各种应用中表现出卓越的性能，但随着模型规模的减小，其对话能力急剧下降，这对其在资源受限环境中的部署构成了障碍。利用直接偏好优化进行知识蒸馏（dDPO）已成为一种有前景的方法，旨在利用较大的教师模型来增强较小模型的对话能力。然而，当前方法主要侧重于“黑盒”知识蒸馏，仅使用教师的响应，忽视了教师提供的输出分布。本文通过引入daDPO（分布感知DPO）来解决这一空白，这是一种统一偏好优化与基于分布的蒸馏的方法。我们提供了严格的理论分析和实证验证，表明daDPO在恢复剪枝模型性能和增强较小LLM模型方面优于现有方法。值得注意的是，在域内评估中，我们的方法使剪枝20%的Vicuna1.5-7B达到接近教师的性能（偏好率-7.3%，而dDPO为-31%），并使Qwen2.5-1.5B偶尔超越其7B教师模型（胜率14.0%）。
英文摘要：
Large language models (LLMs) have demonstrated exceptional performance across various applications, but their conversational abilities decline sharply as model size decreases, presenting a barrier to their deployment in resource-constrained environments. Knowledge distillation with Direct Preference Optimization (dDPO) has emerged as a promising approach to enhancing the conversational abilities of smaller models using a larger teacher model. However, current methods primarily focus on 'black-box' KD, which only uses the teacher's responses, overlooking the output distribution offered by the teacher. This paper addresses this gap by introducing daDPO (Distribution-Aware DPO), a unified method for preference optimization and distribution-based distillation. We provide rigorous theoretical analysis and empirical validation, showing that daDPO outperforms existing methods in restoring performance for pruned models and enhancing smaller LLM models. Notably, in in-domain evaluation, our method enables a 20% pruned Vicuna1.5-7B to achieve near-teacher performance (-7.3% preference rate compared to that of dDPO's -31%), and allows Qwen2.5-1.5B to occasionally outperform its 7B teacher model (14.0% win rate).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
