arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.09472cs.LGcs.AIcs.CL
收藏

从易到难泛化:超越人类监督的可扩展对齐

Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision

Carnegie Mellon University(卡内基梅隆大学);Peking University(北京大学);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室);University of Cambridge(剑桥大学);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);UMass Amherst(马萨诸塞大学阿默斯特分校)

Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, Chuang Gan

AI总结:

本文提出用简单题训练过程监督奖励模型,再以重排序或强化学习指导模型解决更难推理题,在仅使用简单题人类监督下于MATH500取得显著成绩。

摘要:

当前的人工智能对齐方法依赖于人类提供的演示或判断,因此人工智能系统所习得的能力将以人类能力为上限。这就引出了一个富有挑战性的研究问题:当系统的能力已经超过人类水平时,我们如何继续改进这些系统?本文在通过学习较简单任务(例如 MATH 1–3级问题)上的人工标注来解决困难推理任务(例如 MATH 4–5级问题)的背景下回答了这一问题,我们将其称为从易到难泛化。我们的关键洞见是,在较简单任务的监督信号上训练出的评估器(奖励模型),可以被有效用于为较困难任务的候选解答评分,从而促进不同任务难度层级之间的从易到难泛化。基于这一洞见,我们提出了一种新颖的可扩展对齐方法:首先在简单问题(例如1–3级)上训练(过程监督的)奖励模型,随后使用它们评估策略模型在困难问题上的表现。我们表明,来自评估器的这种从易到难泛化,能够通过重排序或强化学习(RL)使生成器实现从易到难泛化。值得注意的是,我们的过程监督7b RL模型和34b模型(reranking@1024)在MATH500上分别取得了34.0%和52.5%的准确率,尽管它们只使用了简单问题上的人类监督。我们的方法为发展出能够突破人类监督前沿的人工智能系统指出了一条有前景的路径。

英文摘要:

Current AI alignment methodologies rely on human-provided demonstrations or judgments, and the learned capabilities of AI systems would be upper-bounded by human capabilities as a result. This raises a challenging research question: How can we keep improving the systems when their capabilities have surpassed the levels of humans? This paper answers this question in the context of tackling hard reasoning tasks (e.g., level 4-5 MATH problems) via learning from human annotations on easier tasks (e.g., level 1-3 MATH problems), which we term as easy-to-hard generalization. Our key insight is that an evaluator (reward model) trained on supervisions for easier tasks can be effectively used for scoring candidate solutions of harder tasks and hence facilitating easy-to-hard generalization over different levels of tasks. Based on this insight, we propose a novel approach to scalable alignment, which firstly trains the (process-supervised) reward models on easy problems (e.g., level 1-3), and then uses them to evaluate the performance of policy models on hard problems. We show that such easy-to-hard generalization from evaluators can enable easy-to-hard generalizations in generators either through re-ranking or reinforcement learning (RL). Notably, our process-supervised 7b RL model and 34b model (reranking@1024) achieves an accuracy of 34.0% and 52.5% on MATH500, respectively, despite only using human supervision on easy problems. Our approach suggests a promising path toward AI systems that advance beyond the frontier of human supervision.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑