从自我辩论中学习：为多智能体辩论准备推理模型
Learning from Self-Debate: Preparing Reasoning Models for Multi-Agent Debate
AI总结：
本文提出SDRL框架，通过自我辩论训练使模型具备独立问题解决能力和多智能体辩论中的多样化推理处理能力，实验表明其在多辩论协议和智能体配置下均能提升多智能体辩论性能和单模型推理能力。
摘要：
大型语言模型（LLM）的推理能力已通过可验证奖励的强化学习（RLVR）显著提升。在测试阶段，通过多智能体辩论（MAD）进行协作推理已成为提升LLM性能的有希望方法。然而，当前RLVR方法通常训练LLM独立解决问题，而没有明确准备它们在辩论中综合和受益于不同推理路径。在本文中，我们提出了自我辩论强化学习（SDRL），一种训练框架，其中模型从自我辩论中学习，使单个LLM具备强大的独立问题解决能力和处理MAD中多样化推理轨迹的能力。给定提示后，SDRL首先采样多个候选解决方案，然后构建具有多样化推理路径的辩论环境，并生成基于此环境的第二轮响应。最后，SDRL联合优化初始和辩论条件响应，产生一个既能作为独立求解器又能作为辩论参与者有效的模型。在多个基础模型和推理基准上的实验表明，SDRL在多种辩论协议和智能体配置下均能提升MAD性能，同时增强单模型推理能力。
英文摘要：
The reasoning abilities of large language models (LLMs) have been substantially improved by reinforcement learning with verifiable rewards (RLVR). At test time, collaborative reasoning through Multi-Agent Debate (MAD) has emerged as a promising approach for enhancing LLM performance. However, current RLVR methods typically train LLMs to solve problems in isolation, without explicitly preparing them to synthesize and benefit from different rationales that arise during debate. In this work, we propose Self-Debate Reinforcement Learning(SDRL), a training framework where models learn from self-debate, equipping a single LLM with both strong standalone problem-solving ability and the capability to process diverse reasoning trajectories in MAD. Given a prompt, SDRL first samples multiple candidate solutions, then constructs a debate context with diverse reasoning paths and generates second-turn responses conditioned on this context. Finally, SDRL jointly optimizes both the initial and debate-conditioned responses, yielding a model that is effective as both a standalone solver and a debate participant. Experiments across multiple base models and reasoning benchmarks show that SDRL consistently improves MAD performance across diverse debate protocols and agent configurations, while simultaneously strengthening single-model reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
