TokenRatio: 通过比率匹配实现原理化的token级偏好优化
TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
AI总结：
本文提出TBPO方法，通过比率匹配恢复token级偏好最优性，改进对齐质量和训练稳定性，并增加输出多样性。
摘要：
直接偏好优化（DPO）是一种广泛使用的无强化学习方法，用于对齐语言模型，但其在完整序列上建模偏好，尽管生成过程由逐token决策驱动。现有token级扩展通常将序列级Bradley-Terry目标分解到时间步，使前缀（状态级）最优性隐含。我们研究如何仅使用标准序列级成对比较恢复token级偏好最优性。我们引入token级Bregman偏好优化（TBPO），提出一个基于前缀的token级Bradley-Terry偏好模型，推导出Bregman散度密度比率匹配目标，该目标扩展了logistic/DPO损失，同时保持由token级模型诱导的最佳策略，并维持DPO-like的简洁性。我们引入两个实例：TBPO-Q，显式学习轻量级状态基线；TBPO-A，通过优势归一化移除基线。在指令跟随、有用性/无害性以及摘要基准上，TBPO相比强序列级和token级基线提高了对齐质量和训练稳定性，并增加了输出多样性。
英文摘要：
Direct Preference Optimization (DPO) is a widely used RL-free method for aligning language models from pairwise preferences, but it models preferences over full sequences even though generation is driven by per-token decisions. Existing token-level extensions typically decompose a sequence-level Bradley-Terry objective across timesteps, leaving per-prefix (state-wise) optimality implicit. We study how to recover token-level preference optimality using only standard sequence-level pairwise comparisons. We introduce Token-level Bregman Preference Optimization (TBPO), which posits a token-level Bradley-Terry preference model over next-token actions conditioned on the prefix, and derive a Bregman-divergence density-ratio matching objective that generalizes the logistic/DPO loss while preserving the optimal policy induced by the token-level model and maintaining DPO-like simplicity. We introduce two instantiations: TBPO-Q, which explicitly learns a lightweight state baseline, and TBPO-A, which removes the baseline through advantage normalization. Across instruction following, helpfulness/harmlessness, and summarization benchmarks, TBPO improves alignment quality and training stability and increases output diversity relative to strong sequence-level and token-level baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
