并非所有偏好都是后训练所需的：面向偏好优化的选择性对齐策略
Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization
AI总结：
针对LLMs后训练对齐中token贡献不均的问题，提出利用策略与参考模型的token级对数概率差异聚焦高影响token的Selective-DPO方法，有效降低开销并提升对齐保真度，实验证明其优于标准DPO和蒸馏基线。
摘要：
大语言模型（LLMs）的后训练对齐是一项关键挑战，因为并非所有token对模型性能的贡献都相同。本文提出了一种选择性对齐策略，利用当前策略与参考模型之间的token级对数概率差异，优先处理偏好对中的高影响token。通过聚焦这些信息丰富的token，我们的方法降低了计算开销并提升了对齐保真度。我们进一步探讨了参考模型质量的作用，证明更强的参考模型能显著提升token选择准确性与整体优化效果。在Arena-Hard和MT-Bench等基准上的综合实验验证了我们的Selective-DPO方法优于标准DPO和基于蒸馏的基线。研究结果突显了token级优化与参考模型选择在推进LLMs偏好对齐中的重要性。代码可在https://github.com/Dongzhijin/SDPO获取。
英文摘要：
Post-training alignment of large language models (LLMs) is a critical challenge, as not all tokens contribute equally to model performance. This paper introduces a selective alignment strategy that prioritizes high-impact tokens within preference pairs, leveraging token-level log-probability differences between the current policy and a reference model. By focusing on these informative tokens, our approach reduces computational overhead and enhances alignment fidelity. We further explore the role of reference model quality, demonstrating that stronger reference models significantly improve token selection accuracy and overall optimization effectiveness. Comprehensive experiments on benchmarks such as Arena-Hard and MT-Bench validate the superiority of our Selective-DPO method over standard DPO and distillation-based baselines. Our findings highlight the importance of token-level optimization and reference model selection in advancing preference alignment for LLMs. The code is available at https://github.com/Dongzhijin/SDPO.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
