AI 中文总结
本研究分离高UTD下SAC评论家的表示坍缩与$|Q|$增长，发现早期$|Q|$增长速率可预测价值发散，据此中止可节省计算，而秩坍缩可恢复。
AI 中文摘要
提高更新-数据比（UTD）会以两种方式破坏离策略评论家，统称为“可塑性损失”：表示坍缩和价值幅度$|Q|$的增长。我们在使用缩放评论家（宽度2048，无归一化）的Soft Actor-Critic（SAC）中将它们分离。坍缩是可存活的：在UTD比率为16时，HalfCheetah评论家中大多数单元处于休眠状态但仍能继续学习，而训练守卫（其停止损失或$|Q|$爆炸的运行）从未标记它们。在一个高UTD SAC配置内，运行开始时彼此接近，而评论家的$\log_{10}|Q|$在步骤15k时攀升的距离，即其早期增长，按守卫标记它们的速度对运行进行排序。在15k时，被标记运行的$|Q|$中位数低于其标记水平一百多倍，然而攀升速率已经对标记进行了排序（在Walker2d上Harrell's C和样本外AUC为0.78，在Ant上为0.98，UTD比率为4）。休眠状态则不然。基于此速率中止可节省约十分之一的保留Walker2d计算量，并保持实时净收益为正。LayerNorm评论家降低了速率，在UTD比率为4时移除了Walker2d上的标记，并降低了回报。
英文摘要
Raising the update-to-data (UTD) ratio breaks off-policy critics in two ways grouped as "plasticity loss": collapsing representations and growing value magnitude $|Q|$. We separate them in Soft Actor-Critic (SAC) with scaled critics (width 2048, no normalization). Collapse is survivable: at UTD ratio 16, HalfCheetah critics with most units dormant keep learning, and the training guard, which stops runs whose loss or $|Q|$ explodes, never flags them. Within one high-UTD SAC configuration, runs start close together, and how far a critic's $\log_{10}|Q|$ has climbed by step 15k, its early growth, ranks the runs by how soon the guard flags them. At 15k, a flagged run's $|Q|$ sits a median of over a hundredfold below its flag level, yet the climb's rate already orders the flags (Harrell's C and out-of-sample AUC 0.78 on Walker2d, 0.98 on Ant, at UTD ratio 4). Dormancy does not. Aborting on this rate saves about a tenth of held-out Walker2d compute and stays net-positive live. A LayerNorm critic lowers the rate, removes the flag on Walker2d at UTD ratio 4 and lowers the return.
Comments31 pages, 9 figures; preprint under review
