行动信用何时需要更新？
When Does Action Credit Need Updating?
- College of Computing（计算学院）
- Georgia Institute of Technology（佐治亚理工学院）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
针对工具型AI代理策略更新后行动信用过时的问题，提出成对分支敏感性与一阶锚定信用传输估计器，并设计决策充分信用门（DSC-Gate）以决定重用、传输或重采样信用，实验表明该方法在保持决策质量的同时大幅减少额外工具调用。
AI中文摘要：
使用工具的人工智能代理会不断用新的交互数据进行更新。然而，在每次策略更新之后，先前估计的行动信用可能变得过时。从头重新计算它们可能需要大量的额外工具调用和环境交互，使得重复更新变得越来越昂贵。我们提出一个简单的问题：历史行动信用究竟何时需要更新？我们的关键观察是，行动价值的变化并不一定意味着决策的变化。只要策略引起的漂移太小，不足以推翻现有的行动排序，历史信用仍然可以是有用的。基于这一想法，我们引入了成对分支敏感性，以捕捉策略更新对区分两个候选行动的下游区域的影响强度。然后，我们推导出一个一阶锚定信用传输估计器，利用旧的干预轨迹来更新历史信用，并提出一个决策充分信用门（DSC-Gate），用于选择是重用、传输还是重新采样信用。实验表明，分支敏感性比全局策略距离能更好地解释信用漂移。在拥有足够历史数据的情况下，信用传输减少了估计误差，而其对决策制定的益处集中在影响区分行动分支的更新上。在一个完全独立的测试集上，DSC-Gate相对于基于差距的门仅将平均遗憾改变了+0.00004，同时将平均新工具步骤从472减少到286，减少了39.4%。在真实工具代理参数更新后，我们观察到相同的模式。总体而言，我们的结果表明，代理不需要在每次策略更新后重新计算行动信用：大部分历史证据可以被重用或廉价地纠正，从而减少了保持行动决策最新所需的额外交互。
英文摘要：
Tool-using agents are continually updated with new interaction data. After each policy update, however, previously estimated action credits may become stale. Recomputing them from scratch can require many additional tool calls and environment interactions, making repeated updates increasingly expensive. We ask a simple question: when does historical action credit actually need to be updated? Our key observation is that a change in action value does not necessarily imply a change in the decision. Historical credit can still be useful as long as policy-induced drift is too small to overturn the existing action ranking. Building on this idea, we introduce pairwise branch sensitivity to capture how strongly a policy update affects the downstream regions that distinguish two candidate actions. We then derive a first-order anchored credit-transport estimator that updates historical credit using old interventional trajectories, and propose a Decision-Sufficient Credit Gate (DSC-Gate) that chooses whether to reuse, transport, or resample credit. Experiments show that branch sensitivity explains credit drift substantially better than global policy distance. With sufficient historical data, credit transport reduces estimation error, while its benefit to decision making is concentrated on updates that affect action-distinguishing branches. On a fully independent test set, DSC-Gate changes mean regret by only +0.00004 relative to a gap-based gate while reducing mean new tool steps from 472 to 286, a 39.4% reduction. We observe the same pattern after a real tool-agent parameter update. Overall, our results show that agents do not need to recompute action credit after every policy update: much of the historical evidence can be reused or cheaply corrected, reducing the additional interaction required to keep action decisions up to date.
