WISE: 加权迭代专家社会用于鲁棒多模态多智能体辩论
WISE: Weighted Iterative Society-of-Experts for Robust Multimodal Multi-Agent Debate
AI总结：
本文提出WISE框架，通过多模态多智能体辩论提升视觉语言推理任务的准确性，实验显示在多个数据集上提升了2-7%的性能。
摘要：
最近的大型语言模型（LLMs）在多样化的语料库和任务上进行训练，使它们发展出互补的优势。多智能体辩论（MAD）已作为一种流行的方式，利用这些优势进行稳健的推理，尽管它主要应用于仅语言的任务，其在多模态问题上的有效性仍被忽视。在本文中，我们研究MAD用于解决视觉和语言推理问题。我们的设置使辩论协议能够泛化，利用具有单模态和多模态能力的异质专家。为此，我们提出了加权迭代专家社会（WISE），一种通用且模块化的MAD框架，将智能体分为解决者，生成解决方案，和反思者，验证正确性，分配权重，并提供自然语言反馈。为了在辩论轮次中聚合智能体的解决方案，同时考虑其响应的方差和反馈权重，我们提出了一种改进的Dawid-Skene算法用于后处理，该算法整合了我们的两阶段辩论模型。我们在SMART-840、VisualPuzzles、EvoChart-QA和一个新的SMART-840++数据集上评估WISE，该数据集包含程序生成的问题实例，难度受控。我们的结果表明，WISE在各种多模态任务和LLM配置中，相对于最先进的MAD设置和聚合方法，始终提高了2-7%的准确性。
英文摘要：
Recent large language models (LLMs) are trained on diverse corpora and tasks, leading them to develop complementary strengths. Multi-agent debate (MAD) has emerged as a popular way to leverage these strengths for robust reasoning, though it has mostly been applied to language-only tasks, leaving its efficacy on multimodal problems underexplored. In this paper, we study MAD for solving vision-and-language reasoning problems. Our setup enables generalizing the debate protocol with heterogeneous experts that possess single- and multi-modal capabilities. To this end, we present Weighted Iterative Society-of-Experts (WISE), a generalized and modular MAD framework that partitions the agents into Solvers, that generate solutions, and Reflectors, that verify correctness, assign weights, and provide natural language feedback. To aggregate the agents' solutions across debate rounds, while accounting for variance in their responses and the feedback weights, we present a modified Dawid-Skene algorithm for post-processing that integrates our two-stage debate model. We evaluate WISE on SMART-840, VisualPuzzles, EvoChart-QA, and a new SMART-840++ dataset with programmatically generated problem instances of controlled difficulty. Our results show that WISE consistently improves accuracy by 2-7% over the state-of-the-art MAD setups and aggregation methods across diverse multimodal tasks and LLM configurations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
