DeliberationBench: 当更多声音有害时?多LLM协商协议的受控研究
DeliberationBench: When Do More Voices Hurt? A Controlled Study of Multi-LLM Deliberation Protocols
AI总结:
DeliberationBench研究发现,多LLM协商协议在性能上远低于简单基线,挑战了复杂性提升质量的假设。
摘要:
在大型语言模型(LLMs)通过协商达成共识的多智能体系统中,其实际价值相较于更简单的方法仍缺乏充分审视。我们介绍了DELIBERATIONBENCH,一个评估三种协商协议相对于从模型输出池中选择最佳响应的强基线的受控基准。在270个问题和三个独立种子(总计810次评估)中,我们发现一个显著的负面结果:最佳单基线实现了82.5%±3.3%的胜率,远超最佳协商协议(13.8%±2.6%)。这种6.0倍的性能差距具有统计显著性(p<0.01),且计算成本高出1.5-2.5倍。我们的发现挑战了多LLM系统中复杂性提升质量的假设。
英文摘要:
Multi-agent systems where Large Language Models (LLMs) deliberate to form consensus have gained significant attention, yet their practical value over simpler methods remains under-scrutinized. We introduce DELIBERATIONBENCH, a controlled benchmark evaluating three deliberation protocols against a strong baseline of selecting the best response from a pool of model outputs. Across 270 questions and three independent seeds (810 total evaluations), we find a striking negative result: the best-single baseline achieves an 82.5% +- 3.3% win rate, dramatically outperforming the best deliberation protocol(13.8% +- 2.6%). This 6.0x performance gap is statistically significant (p < 0.01) and comes at 1.5-2.5x higher computational cost. Our findings challenge assumptions that complexity enhances quality in multi-LLM systems.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。