AlphaToken: 在LLM后训练中解耦适应性与稳定性的路径感知响应令牌估值
AlphaToken: Decoupling Adaptation and Stability for Path-Aware Response Token Valuation in LLM Post-Training
AI总结:
提出AlphaToken框架,通过解耦适应性(促进目标任务学习)和稳定性(保持预训练能力)并引入路径感知机制,利用Fisher漂移代理和Ghost点积扩展实现高效令牌估值,从而在微调和偏好优化中屏蔽低价值令牌,提升后训练性能并缓解灾难性遗忘。
摘要:
令牌选择对于有效的LLM后训练至关重要。然而,现有方法大多依赖局部启发式,很少将令牌选择形式化为对单个响应令牌的原则性估值。我们引入了$\textbf{AlphaToken}$,一个响应令牌估值框架,它将估值解耦为$\textbf{适应性}$(促进目标任务学习)和$\textbf{稳定性}$(保持预训练能力),并通过结合局部令牌梯度的直接路径信号与自回归生成中的下游因果路径信号,使每个目标具有$\textbf{路径感知}$性。由于保留数据通常不可用,AlphaToken通过锚定在预训练参考模型上的$\textbf{Fisher漂移代理}$来近似稳定性。为了高效计算,我们将Ghost点积扩展到令牌级估值。AlphaToken在微调和偏好优化过程中屏蔽低价值响应令牌,将训练信号集中在更有价值的位置。实验表明,AlphaToken提高了后训练性能并缓解了灾难性遗忘。
英文摘要:
Token selection is pivotal for effective LLM post-training. However, existing methods mostly rely on local heuristics and rarely formulate token selection as a principled valuation of individual response tokens. We introduce $\textbf{AlphaToken}$, a response token valuation framework that decouples valuation into $\textbf{adaptation}$ (promoting target-task learning) and $\textbf{stability}$ (preserving pre-trained capabilities), and makes each objective $\textbf{path-aware}$ by combining the direct-path signal from local token gradients with the downstream causal-path signal in autoregressive generation. Since retention data are typically unavailable, AlphaToken approximates stability via a $\textbf{Fisher-drift proxy}$ anchored at the pre-trained reference model. For efficient computation, we extend Ghost Dot-Product to token-level valuation. AlphaToken masks low-value response tokens during fine-tuning and preference optimization, concentrating training signals on more valuable positions. Experiments show that AlphaToken improves post-training performance and mitigates catastrophic forgetting.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。