在Camera Dropbox中扩展MONA:再现、学习批准与奖励黑客缓解的设计影响
Extending MONA in Camera Dropbox: Reproduction, Learned Approval, and Design Implications for Reward-Hacking Mitigation
AI总结:
该研究通过扩展MONA方法,验证了学习批准在缓解奖励黑客中的有效性,并探讨了其设计影响。
摘要:
我的优化与非我的优化(MONA)通过限制智能体的规划时间 horizon 而缓解多步骤奖励黑客问题,同时提供远见的批准作为训练信号。原始论文提出关键问题:批准方法的构建方式,特别是批准对实现结果的依赖程度,如何影响MONA的安全保证。本文提出一个以再现优先的公共MONA Camera Dropbox环境扩展,包括重新包装发布的代码库为标准Python项目,使用释放的参考数组确认普通RL与oracle MONA的对比,以及引入模块化的学习批准套件。在有限预算的试点扫描中,最佳校准的学习监督运行实现了零观察奖励黑客,但比oracle MONA的预期行为率显著低。这些结果将MONA论文的批准-谱猜想转化为可运行的实验对象,并表明核心工程挑战从证明MONA概念转向构建保留足够远见的学习批准模型,而不会重新打开奖励黑客通道。代码、配置和再现命令已公开。
英文摘要:
Myopic Optimization with Non-myopic Approval (MONA) mitigates multi-step reward hacking by restricting the agent's planning horizon while supplying far-sighted approval as a training signal~\cite{farquhar2025mona}. The original paper identifies a critical open question: how the method of constructing approval -- particularly the degree to which approval depends on achieved outcomes -- affects whether MONA's safety guarantees hold. We present a reproduction-first extension of the public MONA Camera Dropbox environment that (i)~repackages the released codebase as a standard Python project with scripted PPO training, (ii)~confirms the published contrast between ordinary RL (91.5\% reward-hacking rate) and oracle MONA (0.0\% hacking rate) using the released reference arrays, and (iii)~introduces a modular learned-approval suite spanning oracle, noisy, misspecified, learned, and calibrated approval mechanisms. In reduced-budget pilot sweeps across approval methods, horizons, dataset sizes, and calibration strategies, the best calibrated learned-overseer run achieves zero observed reward hacking but substantially lower intended-behavior rates than oracle MONA (11.9\% vs.\ 99.9\%), consistent with under-optimization rather than re-emergent hacking. These results operationalize the MONA paper's approval-spectrum conjecture as a runnable experimental object and suggest that the central engineering challenge shifts from proving MONA's concept to building learned approval models that preserve sufficient foresight without reopening reward-hacking channels. Code, configurations, and reproduction commands are publicly available. https://github.com/codernate92/mona-camera-dropbox-repro
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。