动作塑形：策略吸收其所能表达的内容
Action Shaping: Policies Absorb What They Can Express
浏览论文内容
中文总结 AI 辅助
本文提出动作塑形，证明策略能吸收其输出层可精确复现的动作偏移，且移除该偏移不影响回报，为动作塑形提供了理论依据和诊断方法。
中文摘要 AI 辅助
奖励塑形有一个定理：基于势能的项可以在不改变最优策略的情况下被移除。同样的做法在动作通道上，即在训练中添加并在部署时丢弃的偏移量，却没有定理。没有任何东西能抵消动作偏移量，因此修正要么在部署时保留，要么在无保证的情况下被移除。我们称之为动作塑形并阐述其原理。一个可训练的策略会吸收其自身输出层能够精确复现的偏移量，这就是我们所说的“表达”的含义；被吸收的内容可以在保持回报不变的情况下被移除。其最小实例是一个零初始化的线性头，位于一个可学习的门之后，添加到通过学习的动作价值函数训练的actor上，无需惩罚或调度。门会自行上升然后下降，对于确定性和随机性actor均如此，在20个任务上移除该头几乎不造成损失。条件是精确复现，而非容量：具有更多参数的非线性头不会被吸收，在配对对照中，向非线性基础头添加一个线性路径可恢复吸收。精确复现使得损失函数具有一个平坦方向，梯度噪声沿该方向漂移，并且偏移量的幅度在移除之前指示了丢弃该头将带来的代价。因此，动作塑形获得了塑形定理的对应物，即吸收条件，以及其背后的机制和可读取该条件的诊断方法。策略吸收它们所能表达的内容，仅此而已。
英文摘要
Reward shaping has a theorem: a potential-based term can be removed without changing the optimal policy. The same practice on the action channel, an offset added in training and dropped at deployment, has no theorem. Nothing cancels an action offset, so the correction is kept at deployment or removed without a guarantee. We call it action shaping and state its principle. A trainable policy absorbs an offset its own output layer can reproduce exactly, which is what we mean by express; what is absorbed can be removed with the return intact. Its minimal instance is a zero-initialized linear head behind a learnable gate, added to an actor that trains through a learned action-value function, with no penalty or schedule. The gate rises and then falls on its own, for deterministic and stochastic actors alike, and on 20 tasks removing the head costs almost nothing. The condition is exact reproduction, not capacity: a nonlinear head with more parameters is not absorbed, and in a paired control, one linear path added to a nonlinear base head restores absorption. Exact reproduction gives the loss a flat direction that gradient noise drifts along, and the offset's amplitude indicates, before removal, what dropping the head will cost. Action shaping thus gains the counterpart of the shaping theorem, a condition for absorption, together with the mechanism behind it and a diagnostic that reads it. Policies absorb what they can express, and only that.
