多智能体LLM辩论中分歧与回答质量的分层分析
What Does Multi-Agent Debate Actually Change?
浏览论文内容
中文总结 AI 辅助
本研究通过四种测量方法分析多智能体LLM辩论，发现语气显著影响报告的一致性，但辩论主要改变表面言论，对最终答案质量无显著提升。
中文摘要 AI 辅助
多智能体辩论，即多个LLM在回答前交换论点，被广泛认为通过揭示真实分歧来提高回答质量。然而，这一机制很少被检验。我们引入了四种测量方法：(A) 辩论者报告的一致性；(B) 其回复文本是否实际提出反驳；(C) 当引发指令被移除后，立场是否持续；(D) 对于开放权重模型，辩论者自身token对数概率中的立场响应。我们评估了三个模型组成的委员会，在750场辩论中，以三种语气辩论开放式GlobalOpinionQA问题：友好（寻求共识）、中立和敌对（压力测试每个立场）。(A) 语气强烈重塑了报告的一致性：在友好和敌对端点之间，完全一致性的差异为50.4个百分点。(B) 一个仅阅读回复文本、从不阅读自我报告或条件的评判者，恢复了相同的模式。(C) 异议部分似乎与引发它的指令有关：在删除敌对指令后，标签回归一致性的频率比在保留指令的匹配重新提问下高出23.1个百分点；仅基于第一轮回合的问题加权推断不具结论性（p=0.0625），但汇总所有轮次后显著（p=0.016），且28个第一轮回归中只有11个也出现在回复文本中。(D) 反对论点更一致地削弱辩论者的立场边际，而非改变其方向。对于最终答案，我们未检测到质量提升：一个经过偏差检查的评审团返回299/299平局（仅排除较大差异），在可验证的控制任务上的准确性不变，而未经过偏差检查的评审团有66%的时间宣布辩论获胜——这是阅读顺序的产物。综合来看，LLM辩论容易改变智能体所说的内容，但我们发现较弱的证据表明它改变了智能体持续支持的内容或提高了最终答案的质量。
英文摘要
Multi-agent debate, in which several LLMs exchange arguments before producing an answer, raises a basic question: does expressed disagreement reflect changes in the members' own positions? No single signal can settle this question, so we organize the analysis around five questions: (A) does the debater say it disagrees; (B) does its reply text actually argue; (C) does its own position change after each debate turn; (D) how much, quantitatively, does the position change; and (E) how do members' final positions compare with their initial ones? We evaluate two- and three-member committees on 50 curated opinion questions from GlobalOpinionQA, using same-model, same-family, and mixed-family configurations with friendly, neutral, and hostile instructions assigned at each turn. (A) Tone changes reported agreement: the share of replies reporting strong agreement is 70.7-96.3% under friendly instructions, compared with 9.5-19.3% under hostile ones, varying with model choice. (B) Self-reports broadly align with text judgments, but consistency varies by agreement level and model choice. (C) Position changes depend on the interaction: after a peer's leaning-disagree reply, members reporting strong agreement switch options more often than those reporting leaning disagreement. (D) For open-weight members, the probability of the option a member already holds stays near saturation, even after a peer's pushback, while endorsement of its earlier position text drops after a peer's argument relative to neutral filler, more for Qwen3.8-27B than Inkling. (E) The selected option is unchanged from members' initial to final positions in over 90% of comparisons in every configuration. Taken together, expressed disagreement need not translate into position revision, either within individual exchanges or over a complete debate.
