Avoiding Tampering Incentives in Deep RL via Decoupled Approval

REALab: An Embedded Perspective on Tampering

This paper describes REALab, a platform for embedded agency research in reinforcement learning (RL). REALab is designed to model the structure of tampering problems that may arise in real-world deployments of RL. Standard Markov Decision Process (MDP) formulations of RL and simulated environments mirroring the MDP structure assume secure access to feedback (e.g., rewards). This may be unrealistic in settings where agents are embedded and can corrupt the processes producing feedback (e.g., human supervisors, or an implemented reward function). We describe an alternative Corrupt Feedback MDP for…

MONA：近视优化与非近视批准可缓解多步奖励黑客行为

MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hacking

MONA方法通过结合短视优化与远视奖励，在无需检测奖励黑客或额外信息的情况下，有效防止多步奖励黑客行为，并在多种错位场景中验证了其有效性。

Approval Voting and Incentives in Crowdsourcing

The growing need for labeled training data has made crowdsourcing an important part of machine learning. The quality of crowdsourced labels is, however, adversely affected by three factors: (1) the workers are not experts; (2) the incentives of the workers are not aligned with those of the requesters; and (3) the interface does not allow workers to convey their knowledge accurately, by forcing them to make a single choice among a set of options. In this paper, we address these issues by introducing approval voting to utilize the expertise of workers who have partial knowledge of the true answe…