ARCA: 当令牌信号退化时的适配器-残差信用分配
ARCA: Adapter-Residual Credit Assignment When Token Signals Degenerate
AI总结：
针对LoRA微调下令牌级信用分配信号退化的问题，提出ARCA方法，利用适配器隐藏状态残差作为令牌显著性度量，无需学习奖励模型或价值头。
摘要：
语言模型强化学习的令牌级信用分配通常被表述为策略完全可训练，而实际的LLM-RL流程往往依赖于参数高效微调，尤其是LoRA。我们认为这种分离隐藏了一种结构性失效模式。在LoRA下，策略被限制在参考模型的低秩邻域内，因此常用内在信用信号（如惊奇度、熵减和策略散度）所依赖的每令牌输出分布差异，在轨迹内归一化后可能变得退化，要么接近均匀权重，要么集中在少量与任务无关的位置上。我们形式化了这种行为，并提出直接用浓度诊断指标（如权重基尼系数和有效令牌比率）进行测量。然后，我们引入了适配器-残差信用分配（ARCA），一种轻量级替代方案，它从适配器自身的隐藏状态残差 $\|h^{\text{adapted}}_t - h^{\text{base}}_t\|_2$ 中推导令牌显著性。ARCA关注适配器实际改变模型的位置，而不是输出分布显得不确定或偏移的位置，并且不需要学习奖励模型、价值头或树结构。在紧凑的MATH/Qwen3-1.7B GRPO扫描中，ARCA在匹配的轨迹预算下表现出预测的非退化中间区域信用分布，并与秩匹配的基线保持竞争力。
英文摘要：
Token-level credit assignment for language-model reinforcement learning is usually formulated as if the policy were fully trainable, while practical LLM-RL pipelines often rely on parameter-efficient fine-tuning, especially LoRA. We argue that this separation hides a structural failure mode. Under LoRA, the policy is restricted to a low-rank neighborhood of the reference model, so the per-token output-distribution differences used by common intrinsic credit signals, surprisal, entropy reduction, and policy divergence, can become degenerate after within-trajectory normalization, either approaching uniform weights or concentrating on a small set of task-agnostic positions. We formalize this behavior and propose measuring it directly with concentration diagnostics such as weight Gini and effective-token ratio. We then introduce \emph{Adapter-Residual Credit Assignment} (ARCA), a lightweight alternative that derives token salience from the adapter's own hidden-state residual, $\|h^{\text{adapted}}_t - h^{\text{base}}_t\|_2$. ARCA asks where the adapter actually changes the model, rather than where the output distribution appears uncertain or shifted, and requires no learned reward model, value head, or tree construction. In a compact MATH/Qwen3-1.7B GRPO sweep, ARCA exhibits the predicted non-degenerate middle-regime credit distribution under matched rollout budgets and remains competitive with rank-matched baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
