在SAC中解除tanh雅可比限制：关于Bang-Bang控制与MetaDrive的一个负面结果
Unthrottling the Tanh Jacobian in SAC: A Negative Result on Bang-Bang Control and MetaDrive
浏览论文内容
中文总结 AI 辅助
本文通过最小干预测试发现，在SAC中恢复tanh雅可比消失的信号（旁路项）在bang-bang控制和MetaDrive任务上均无益，反而导致策略饱和和性能下降，表明雅可比效应不应被当作需修复的缺陷。
中文摘要 AI 辅助
Soft Actor-Critic (SAC) 将连续策略表示为经tanh压缩的无界高斯分布。该映射的雅可比为 $\partial a/\partial u = 1-a^2$，当 $|a|\to 1$ 时趋于零。一个自然的担忧是，这种节流效应恰好在极端动作（全刹车、全油门）最优的情况下削弱了演员网络所获得的评论家信号。我们测试了一种恢复缺失信号的最小干预措施：在演员损失中增加一项，其相对于tanh前均值的梯度是 $Q$ 的分离动作梯度，且无增益参数。在最小时间双积分器上（其最优解为动作边界处的bang-bang控制），普通SAC在十对配对种子下已达到接近最优的回报（$-31.6$ 对比校准最优值 $-30.3$）。无门控的旁路确实使策略饱和（99%的评估步骤中 $|a|\ge 0.9$），并将回报降至 $-195.5$。仅在平坦肩部 $|a|\in[0.9,0.999]$ 触发的门控旁路同样失败，且未留下饱和策略。热启动的MetaDrive微调显示出相同模式：旁路并未改善回报，且碰撞率下降之处通常以驶出道路为代价。自动调节的熵系数随旁路而上升，这是向尾部的推动。雅可比效应是真实的。将其视为需要修复的缺陷并非没有代价，在本文研究的任务上并无帮助。使边界饱和并不等同于解决最优值位于该边界上的问题。
英文摘要
Soft Actor-Critic (SAC) represents a continuous policy as an unbounded Gaussian that is squashed by tanh. The Jacobian of that map is $\partial a/\partial u = 1-a^2$, which vanishes as $|a|\to 1$. A natural concern is that this throttle starves the actor of critic signal exactly where extreme actions (full brake, full throttle) are optimal. We test a minimal intervention that restores the missing signal: one extra term in the actor loss whose gradient on the pre-tanh mean is the detached action-gradient of $Q$, with no gain parameter. On a minimum-time double integrator whose optimum is bang-bang at the action bounds, vanilla SAC already reaches near-optimal return ($-31.6$ vs. a calibrated optimum of $-30.3$) across ten paired seeds. An ungated bypass does saturate the policy (99% of eval steps with $|a|\ge 0.9$) and collapses return to $-195.5$. A gated bypass that fires only on the flat shoulder $|a|\in[0.9,0.999]$ also fails, and does so without leaving a saturated policy. Warm-started MetaDrive fine-tuning shows the same pattern: the bypass does not improve return, and where collision rate falls it is typically traded for out-of-road departures. Auto-tuned entropy coefficient rises against the bypass, which is a push toward the tails. The Jacobian effect is real. Treating it as a bug to be undone is not free, and on the tasks studied here it is not helpful. Saturating a bound is not the same as solving a problem whose optimum lives on that bound.
发表机构
- Independent Researcher（独立研究者）
机构由 AI 辅助整理，请以论文原文为准。
