ProcGen 泛化差距衡量的是什么？动作规则、收敛性与缺失的随机基线
What Does a ProcGen Generalization Gap Measure? Action Rules, Residual Entropy, and the Missing Random Floor
AI总结：
本研究指出强化学习泛化差距报告缺失随机基线，通过 ProcGen 实验和代码审计揭示动作规则与收敛性影响，建议标准化报告协议。
AI中文摘要：
强化学习中的泛化差距（训练关卡上的回报减去保留关卡上的回报）通常在没有参考点的情况下报告。我们认为缺失的参考是一个实测的随机基线：即在相同关卡和相同测试环境下，均匀随机策略的回报。在 ProcGen 上，该基线改变了几项标准数值的含义。在八个环境的相同检查点和关卡上，在采样和贪心（argmax）测试时动作之间切换会使保留关卡上的回报向两个方向移动，且贪心评估使三个环境降至或低于基线：在 miner 中，采样策略在保留关卡上的得分是基线的 4.9 倍，而其 argmax 得分低于基线。原始策略熵显示八个环境中有七个未达到收敛，但其中 35-65% 的熵分布在具有相同效果的动作上；合并这些动作后，仅剩一到三个环境未收敛，且相对于基线，只有 heist 没有学到任何可迁移的东西。对十二个先前 ProcGen 代码库的审计发现，所有十一个具有保留评估的代码库对其策略梯度智能体采样测试时动作，其中九个是默认行为而非显式选择，六个报告运行循环内平均值而非评估固定检查点。应用于我们自己的案例研究，同样的检查降低了一个统计显著的编码器效应，并排除了编码器内训练与测试的 CKA 统计量。我们建议每个报告的差距应说明其动作规则，使用匹配且带种子的协议，并在两个关卡集上报告随机基线。
英文摘要：
A generalization gap in reinforcement learning, return on training levels minus return on held-out levels, is usually reported without a reference point. We argue that it should be read against a measured random floor: the return of a uniform-random policy on the same levels under the same evaluation harness. On eight ProcGen environments with PPO at a compute-limited budget (8M steps, 16 parallel environments; three games extended to 25M), the floor changes what standard numbers mean. The test-time action rule decides which policy is measured: in miner, the sampled policy scores 5.1x the floor on held-out levels while its argmax scores below it in every run, and greedy evaluation places two environments significantly below the floor. Used as a convergence diagnostic, raw policy entropy flags six of eight environments, but 32-66% of that entropy lies on actions with identical effects; against the floor, five of eight sampled policies are clearly above it on held-out levels and heist's is not distinguishable from it. An audit of twelve ProcGen codebases finds that nine sample test-time actions with no explicit choice at the evaluation call site. We recommend that every reported gap state its action rule, seed its evaluation and specify its tests before analysis, and report the floor on both level sets.
