利用多智能体分歧进行多模态推理中的工具招募
DART: Leveraging Multi-Agent Disagreement for Tool Recruitment in Multimodal Reasoning
AI总结：
DART通过多智能体分歧识别有用视觉工具，提升多模态推理中的工具调用效果。
摘要：
专门的视觉工具可以增强大语言模型或视觉语言模型，使其具备专家知识（例如，定位、空间推理、医学知识等），但知道何时调用哪些工具具有挑战性。我们介绍了DART，一种多智能体框架，利用多个辩论视觉智能体之间的分歧来识别有用的视觉工具（例如，目标检测、OCR、空间推理等）以解决智能体之间的分歧。这些工具通过引入新信息并提供与工具对齐的同意分数来促进富有成效的多智能体讨论，从而突出与专家工具一致的智能体。我们利用一个聚合智能体，通过提供智能体输出和工具信息来选择最佳答案。我们在四个多样化的基准上测试了DART，并证明我们的方法优于多智能体辩论以及单智能体工具调用框架，在A-OKVQA和MMMU上分别优于多智能体辩论（带法官模型）的基线方法3.4%和2.4%。我们还发现DART能够很好地适应应用领域的新工具，在M3D医学数据集上比其他强大的工具调用、单智能体和多智能体基线方法提高了1.3%。此外，我们通过测量轮次间的文本重叠来突出DART相比现有多智能体方法的丰富讨论。最后，我们研究了工具调用分布，发现多样化的工具可靠地用于帮助解决分歧。
英文摘要：
Specialized visual tools can augment large language models or vision language models with expert knowledge (e.g., grounding, spatial reasoning, medical knowledge, etc.), but knowing which tools to call (and when to call them) can be challenging. We introduce DART, a multi-agent framework that uses disagreements between multiple debating visual agents to identify useful visual tools (e.g., object detection, OCR, spatial reasoning, etc.) that can resolve inter-agent disagreement. These tools allow for fruitful multi-agent discussion by introducing new information, and by providing tool-aligned agreement scores that highlight agents in agreement with expert tools, thereby facilitating discussion. We utilize an aggregator agent to select the best answer by providing the agent outputs and tool information. We test DART on four diverse benchmarks and show that our approach improves over multi-agent debate as well as over single agent tool-calling frameworks, beating the next-strongest baseline (multi-agent debate with a judge model) by 3.4% and 2.4% on A-OKVQA and MMMU respectively. We also find that DART adapts well to new tools in applied domains, with a 1.3% improvement on the M3D medical dataset over other strong tool-calling, single agent, and multi-agent baselines. Additionally, we measure text overlap across rounds to highlight the rich discussion in DART compared to existing multi-agent methods. Finally, we study the tool call distribution, finding that diverse tools are reliably used to help resolve disagreement.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
