泛化类比：将AI监管泛化到难以衡量领域的测试平台
Generalization Analogies: A Testbed for Generalizing AI Oversight to Hard-To-Measure Domains
AI总结：
针对AI系统可能利用人类反馈缺陷投机取巧的风险，该研究构建69种分布偏移探究奖励模型泛化规律，发现奖励模型默认不评估指令遵循，内部表征解读技术泛化性优于微调仍存缺陷，最终整合出GENIES基准以推动奖励模型泛化控制研究。
摘要：
随着AI系统变得更加智能，其行为评估难度也不断提升，它们可能学会利用人类反馈的缺陷投机取巧，而非真正努力遵循指令；不过，通过控制大语言模型（LLM）将人类反馈泛化到反馈不可靠场景的方式，这种风险可以得到缓解。为更好地理解奖励模型的泛化机制，我们构建了涵盖8个类别的69种分布偏移。研究发现，奖励模型默认不会学习评估“指令遵循”能力，反而更偏好类似互联网文本的人设。用于解读奖励模型内部表征的技术比标准微调的泛化效果更好，但仍经常无法区分指令遵循行为与混淆行为。我们将15种最具挑战性的分布偏移整合为GENeralization analogIES（GENIES，泛化类比）基准，希望借此推动奖励模型泛化控制领域的研究进展。
英文摘要：
As AI systems become more intelligent and their behavior becomes more challenging to assess, they may learn to game the flaws of human feedback instead of genuinely striving to follow instructions; however, this risk can be mitigated by controlling how LLMs generalize human feedback to situations where it is unreliable. To better understand how reward models generalize, we craft 69 distribution shifts spanning 8 categories. We find that reward models do not learn to evaluate `instruction-following' by default and instead favor personas that resemble internet text. Techniques for interpreting reward models' internal representations achieve better generalization than standard fine-tuning, but still frequently fail to distinguish instruction-following from conflated behaviors. We consolidate the 15 most challenging distribution shifts into the GENeralization analogIES (GENIES) benchmark, which we hope will enable progress toward controlling reward model generalization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
