优化更宽而非更深：政策优化的共识聚合
Optimize Wider, Not Deeper: Consensus Aggregation for Policy Optimization
AI总结：
本文提出CAPO方法，通过增加模型宽度而非深度来提升政策优化效果，在连续控制任务中表现优于PPO及更深层基线模型。
摘要：
近端策略优化（PPO）通过多个剪裁的SGD周期近似信任区域更新。每个周期可能偏离自然梯度方向，产生路径依赖噪声。本文利用Fisher信息几何学将策略更新分解为信号（自然梯度投影）和浪费（Fisher正交残差）。实验表明，信号饱和但浪费增长，导致优化深度困境。本文提出CAPO，通过K个PPO复制在相同批次上优化，仅改变mini-batch洗牌顺序后聚合。在欧几里得参数空间和策略分布的自然参数空间（通过对数意见池）研究聚合。在自然参数空间，共识在KL惩罚的替代器和信任区域合规性上优于均值专家；参数平均继承这些保证。在连续控制任务中，CAPO在固定样本预算下优于PPO和计算匹配的更深层基线模型，提升达8.6倍。CAPO证明通过优化更宽而非更深层可以提升策略优化，无需额外环境交互。
英文摘要：
Proximal policy optimization (PPO) approximates the trust region update using multiple epochs of clipped SGD. Each epoch may drift further from the natural gradient direction, creating path-dependent noise. To understand this drift, we can use Fisher information geometry to decompose policy updates into signal (the natural gradient projection) and waste (the Fisher-orthogonal residual that consumes trust region budget without first-order surrogate improvement). Empirically, signal saturates but waste grows with additional epochs, creating an optimization-depth dilemma. We propose Consensus Aggregation for Policy Optimization (CAPO), which redirects compute from depth to width: $K$ PPO replicates are optimized on the same batch, differing only in minibatch shuffling order, and then aggregated into a consensus. We study aggregation in two spaces: Euclidean parameter space, and the natural parameter space of the policy distribution via the logarithmic opinion pool. In natural parameter space, the consensus provably achieves higher KL-penalized surrogate and tighter trust region compliance than the mean expert; parameter averaging inherits these guarantees approximately. On continuous control tasks, CAPO outperforms PPO and compute-matched deeper baselines under fixed sample budgets by up to 8.6x. CAPO demonstrates that policy optimization can be improved by optimizing wider, rather than deeper, without additional environment interactions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
