知识分歧与辩论在可扩展监督中的价值
Knowledge Divergence and the Value of Debate for Scalable Oversight
AI总结：
本文通过几何方法分析辩论在可扩展监督中的价值，揭示了辩论与RLAIF之间的联系，并探讨了知识分歧对辩论效果的影响。
摘要：
通过辩论和基于AI反馈的强化学习（RLAIF）都是可扩展监督高级AI系统的方法，但尚无正式框架将它们联系起来或表征辩论何时具有优势。我们通过参数化辩论价值来分析这一点，通过辩论模型之间知识分歧的几何学来表征。利用模型表示子空间之间主角度，我们证明了辩论优势具有精确的闭合形式。当模型共享相同的训练语料库时，辩论退化为RLAIF-like，其中单代理方法恢复相同的最优解。当模型具有分歧的知识时，辩论优势随相变从二次 regime（辩论提供微不足道的收益）到线性 regime（辩论是必要的）而变化。我们分类了三种知识分歧的 regime（共享、单侧和组合）并提供了存在性结果，表明辩论可以实现单个模型无法达到的成果，同时提供了一个负结果，表明足够强的对抗性激励会导致组合 regime 中的协调失败，并有一个明确的阈值将有效的辩论与无效的辩论区分开来。我们提供了辩论与RLAIF之间的第一个正式联系，为理解何时对抗性监督协议是合理的提供了几何基础，并与跨模型互补信息中提取潜在知识的问题建立了联系。
英文摘要：
AI safety via debate and reinforcement learning from AI feedback (RLAIF) are both proposed methods for scalable oversight of advanced AI systems, yet no formal framework relates them or characterizes when debate offers an advantage. We analyze this by parameterizing debate's value through the geometry of knowledge divergence between debating models. Using principal angles between models' representation subspaces, we prove that the debate advantage admits an exact closed form. When models share identical training corpora, debate reduces to RLAIF-like where a single-agent method recovers the same optimum. When models possess divergent knowledge, debate advantage scales with a phase transition from quadratic regime (debate offers negligible benefit) to linear regime (debate is essential). We classify three regimes of knowledge divergence (shared, one-sided, and compositional) and provide existence results showing that debate can achieve outcomes inaccessible to either model alone, alongside a negative result showing that sufficiently strong adversarial incentives cause coordination failure in the compositional regime, with a sharp threshold separating effective from ineffective debate. We offer the first formal connection between debate and RLAIF, a geometric foundation for understanding when adversarial oversight protocols are justified, and connection to the problem of eliciting latent knowledge across models with complementary information.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
