基于注意力的令牌加权直接偏好优化
Token-weighted Direct Preference Optimization with Attention
AI总结：
提出Token-weighted DPO (TwDPO)方法，利用注意力机制估计令牌权重，在不增加额外训练成本的情况下提升大语言模型与人类偏好对齐的性能。
摘要：
直接偏好优化（DPO）无需单独奖励模型即可使大语言模型与人类偏好对齐。然而，DPO平等对待响应中的所有令牌，忽略了单个令牌的不同重要性。现有的令牌级PO方法要么使用基于令牌位置的启发式函数，要么使用单独训练模型给出的概率估计来计算令牌权重，这缺乏鲁棒性且增加了额外训练成本。相比之下，我们提出令牌加权DPO（TwDPO）——一种基于令牌加权RL的新型训练目标，以及AttentionPO——TwDPO的一个实例，它利用LLM自身的注意力来估计令牌权重。AttentionPO提示LLM作为成对评判者，并在比较响应时检查模型关注的位置。这种设计使AttentionPO具有内容感知能力，根据响应内容调整权重，并且高效，每个样本仅需额外两次前向传播。实验结果表明，AttentionPO在AlpacaEval、MT-Bench和ArenaHard上显著提升了性能，超越了现有的偏好优化方法。
英文摘要：
Direct Preference Optimization (DPO) aligns Large Language Models with human preferences without the need for a separate reward model. However, DPO treats all tokens in responses equally, neglecting the differing importance of individual tokens. Existing token-level PO methods compute the token weights using either token-position-based heuristic functions or probability estimates given by a separately trained model, which lacks robustness and incurs extra training cost. In contrast, we propose Token-weighted DPO (TwDPO) -- a novel training objective grounded on token-weighted RL -- and AttentionPO -- an instantiation of TwDPO that uses attention from the LLM itself to estimate token weights. AttentionPO prompts the LLM to serve as a pairwise judge and check where the model attends when comparing the responses. This design makes AttentionPO content-aware, adjusting weights based on response content, and efficient, incurring only two extra forward passes per example. Experiment results show that AttentionPO significantly improves performance on AlpacaEval, MT-Bench, and ArenaHard, surpassing existing Preference Optimization methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
