arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.13124cs.CLcs.AI
收藏

辩论有助于弱到强泛化

Debate Helps Weak-to-Strong Generalization

Tongyi Lab(通义实验室)

Hao Lang, Fei Huang, Yongbin Li

AI总结:

本文结合可扩展监督与弱到强泛化,通过辩论让弱模型从强模型提取可信信息,并用集成弱模型提升监督稳健性,实验证明该方法能改善对齐。

摘要:

用于使已具备能力的模型与期望行为对齐的常见方法依赖于人类提供监督的能力。然而,未来的超人类模型将超越人类的能力。因此,人类将只能对超人类模型进行弱监督。这种预期的人类评估缺陷会削弱未来人工智能系统的安全性。可扩展监督和弱到强泛化是解决这一问题的两种互补方法。在本文中,我们尝试结合这两种方法的优势以进一步改进对齐。具体而言,我们研究如何利用一个强大的预训练模型来改进人类监督,然后用增强的弱人类监督来监督强模型。为了进行迭代的经验性进展,我们考虑一个类比:我们能否使用一个强模型来改进弱模型的监督,然后用它来监督强模型?我们通过在一个小弱模型上使用真实标签进行微调,并借助一个大型强模型的额外帮助,然后使用弱模型生成的标签对强模型进行微调来实证测试这一点。我们发现,辩论可以帮助弱模型从不可信的强模型中提取可信信息,这为在训练弱模型时提供上下文杠杆。我们还表明,弱模型的集成有助于利用强模型辩论者生成的长参数,并获得更稳健的监督估计。在OpenAI弱到强NLP基准上的大量实验表明,组合方法带来了更好的对齐,这表明辩论有潜力帮助弱到强泛化。

英文摘要:

Common methods for aligning already-capable models with desired behavior rely on the ability of humans to provide supervision. However, future superhuman models will surpass the capability of humans. Therefore, humans will only be able to weakly supervise superhuman models. This expected deficiency of human evaluation would weaken the safety of future AI systems. Scalable oversight and weak-to-strong generalization are two complementary approaches to tackle this issue. In this paper, we attempt to combine the strengths of these two approaches to further improve alignment. Specifically, we investigate ways of improving human supervision with a strong pretrained model and then supervise the strong model with enhanced weak human supervision. To make iterative empirical progress, we consider an analogy: can we use a strong model to improve weak model supervision and then use it to supervise the strong model? We empirically test it by finetuning a small weak model on ground truth labels with the additional help from a large strong model, and then finetuning the strong model on labels generated by the weak model. We find that debate can assist a weak model in extracting trustworthy information from an untrustworthy strong model, which provides leverage as context on samples when training a weak model. We also show that an ensemble of weak models helps exploit long arguments generated by strong model debaters and obtain a more robust supervision estimate. Extensive experiments on the OpenAI weak-to-strong NLP benchmarks show that the combination approach leads to better alignment, which indicates that debate has the potential to help weak-to-strong generalization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑