arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.24014cs.AIcs.CL
收藏

强化学习迈向广泛且持久有益的模型

Reinforcement Learning Towards Broadly and Persistently Beneficial Models

OpenAI

Akshay V. Jagadeesh, Rahul K. Arora, Khaled Saab, Ali Malik, Mikhail Trofimov, Foivos Tsimpourlas, Johannes Heidecke, Karan Singhal

AI总结:

通过真实领域中的有益行为强化学习,模型在超过80%的分布外基准上表现提升,并展现出跨领域对齐泛化与对抗攻击的持久性。

摘要:

随着AI系统在日益多样化和高风险场景中部署,模型对齐必须泛化到训练期间未见过的任务和领域。这对于强化学习(RL)尤为重要,因为它可能通过奖励黑客、欺骗或其他意外策略引入意外的对齐问题。我们研究在真实领域中实例化的有益行为RL能否产生超出训练分布的广泛且持久的对齐泛化。我们构建了一个包含真实情境的数据集,旨在衡量和训练有益特质,如诚实、公平、风险意识和可纠正性,涵盖健康、科学和教育等多个领域。然后,我们使用RL在该数据集上训练模型,并在超过50个独立的对齐和有益行为基准上进行评估。与计算匹配的基线相比,有益特质RL在超过80%的这些分布外基准上提升了性能。我们观察到显著的分布外对齐迁移:一个完全局限于健康领域的有益行为RL干预,在非健康对齐评估上产生了广泛改进,包括减少奖励黑客、欺骗和一般性对齐问题。最后,我们研究对齐持久性:在试图引导模型走向对齐失败时,行为是否保持稳健对齐。经过有益特质RL训练的模型显示出改进的持久性,包括对对抗性提示和有害微调更强的抵抗力;需要进一步的工作来隔离这些效应的来源。这些结果表明,在真实领域中强化有益行为的RL可以产生更稳健地与人类繁荣对齐的模型。

英文摘要:

As AI systems are deployed across increasingly diverse and high-stakes settings, model alignment must generalize beyond the tasks and domains seen during training. This is especially important for reinforcement learning (RL), which can introduce unexpected misalignment through reward hacking, deception, or other unintended strategies. We study whether RL on beneficial behavior, instantiated in realistic domains, can produce broad and persistent alignment generalization beyond the training distribution. We construct a dataset of realistic situations designed to measure and train beneficial traits, such as truthfulness, fairness, risk awareness, and corrigibility, spanning varied domains, including health, science, and education. We then train models with RL on this dataset and evaluate them on more than 50 independent benchmarks of alignment and beneficial behavior. Compared to a compute-matched baseline, beneficial trait RL improves performance on over 80% of these out-of-distribution benchmarks. We observe substantial out-of-distribution alignment transfer: a beneficial-behavior RL intervention entirely limited to one domain, health, produces broad improvements on non-health alignment evaluations, including reduced reward hacking, deception, and general misalignment. Finally, we study alignment persistence: whether behavior remains robustly aligned under attempts to steer models towards misalignment. Models trained with beneficial trait RL show improved persistence, including greater resistance to adversarial prompting and harmful finetuning; further work is required to isolate the sources of these effects. These results suggest that RL to reinforce beneficial behavior in realistic domains can produce models that are more robustly aligned with human flourishing.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑