与更具说服力的LLM辩论能得出更真实的答案
Debating with More Persuasive LLMs Leads to More Truthful Answers
AI总结：
研究较弱模型能否通过辩论评估较强模型的正确性，发现以无监督方式优化专家辩手说服力能提升非专家识别真相的能力，为缺乏真实标签下通过辩论对齐模型提供了经验证据。
摘要：
将大型语言模型（LLM）与期望行为对齐的常用方法严重依赖人类标注数据。然而，随着模型日益复杂，它们将超越人类专业知识，人类评估的角色将演变为非专家监督专家。有鉴于此，我们提出问题：较弱的模型能否评估较强模型的正确性？我们在类似设定下研究此问题，其中较强模型（专家）拥有回答问题所需的信息，而较弱模型（非专家）缺乏此信息。我们评估的方法是辩论，即两个LLM专家各自为不同答案辩护，由非专家选择答案。我们发现，辩论持续帮助非专家模型和人类回答问题，分别达到76%和88%的准确率（朴素基线分别为48%和60%）。此外，以无监督方式优化专家辩手的说服力，提升了非专家在辩论中识别真相的能力。我们的结果为在缺乏真实标签的情况下通过辩论对齐模型提供了令人鼓舞的经验证据。
英文摘要：
Common methods for aligning large language models (LLMs) with desired behaviour heavily rely on human-labelled data. However, as models grow increasingly sophisticated, they will surpass human expertise, and the role of human evaluation will evolve into non-experts overseeing experts. In anticipation of this, we ask: can weaker models assess the correctness of stronger models? We investigate this question in an analogous setting, where stronger models (experts) possess the necessary information to answer questions and weaker models (non-experts) lack this information. The method we evaluate is debate, where two LLM experts each argue for a different answer, and a non-expert selects the answer. We find that debate consistently helps both non-expert models and humans answer questions, achieving 76% and 88% accuracy respectively (naive baselines obtain 48% and 60%). Furthermore, optimising expert debaters for persuasiveness in an unsupervised manner improves non-expert ability to identify the truth in debates. Our results provide encouraging empirical evidence for the viability of aligning models with debate in the absence of ground truth.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
