通过可扩展监督与集成学习提升弱到强泛化能力
Improving Weak-to-Strong Generalization with Scalable Oversight and Ensemble Learning
AI总结：
本研究在弱到强泛化框架下模拟超级对齐两阶段，结合可扩展监督与集成学习提升弱监督质量，并通过SciQ任务初步验证第一阶段方法，为超级对齐实证研究提供新路径。
摘要：
本文是对OpenAI近期关于弱到强泛化（W2SG）的超级对齐工作的后续研究。超级对齐旨在确保高级AI系统在处理复杂、高风险任务时，始终与人类价值观和意图保持一致。W2SG框架为这一不断发展的领域的实证研究开辟了新的可能性。\n本研究在W2SG框架下模拟了超级对齐的两个阶段：通用超人类模型的开发阶段和向超级智能演进的阶段。在第一阶段，以人类监督为基础，通过结合可扩展监督与集成学习提升弱监督的质量，缩小弱教师模型与强学生模型之间的能力差距。在第二阶段，采用自动对齐评估器作为弱监督者，通过递归更新该自动对齐器，同步提升弱教师模型的能力，实现对更强学生模型的弱到强监督。\n我们还对第一阶段提出的方法进行了初步验证。以SciQ任务为例，我们通过bagging（自助聚合）和boosting（提升）探索了弱教师模型的集成学习；通过人机交互和AI-AI辩论两种辅助设置探索了可扩展监督。此外，本文还讨论了改进后的弱监督对基于上下文学习的弱到强泛化能力提升的影响。实验代码和数据集将发布于https://github.com/ADaM-BJTU/W2SG。
英文摘要：
This paper presents a follow-up study to OpenAI's recent superalignment work on Weak-to-Strong Generalization (W2SG). Superalignment focuses on ensuring that high-level AI systems remain consistent with human values and intentions when dealing with complex, high-risk tasks. The W2SG framework has opened new possibilities for empirical research in this evolving field. Our study simulates two phases of superalignment under the W2SG framework: the development of general superhuman models and the progression towards superintelligence. In the first phase, based on human supervision, the quality of weak supervision is enhanced through a combination of scalable oversight and ensemble learning, reducing the capability gap between weak teachers and strong students. In the second phase, an automatic alignment evaluator is employed as the weak supervisor. By recursively updating this auto aligner, the capabilities of the weak teacher models are synchronously enhanced, achieving weak-to-strong supervision over stronger student models.We also provide an initial validation of the proposed approach for the first phase. Using the SciQ task as example, we explore ensemble learning for weak teacher models through bagging and boosting. Scalable oversight is explored through two auxiliary settings: human-AI interaction and AI-AI debate. Additionally, the paper discusses the impact of improved weak supervision on enhancing weak-to-strong generalization based on in-context learning. Experiment code and dataset will be released at https://github.com/ADaM-BJTU/W2SG.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
