arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.07553cs.AI
收藏

COMMA:一种基于通信的多模态多智能体基准

COMMA: A Communicative Multimodal Multi-Agent Benchmark

Department of Computer Sciences University of Wisconsin-Madison(计算机科学系威斯康星大学麦迪逊分校);Department of Computer Sciences UC San Diego(计算机科学系加州大学圣地亚哥分校);Department of Radiology University of Wisconsin-Madison(放射学系威斯康星大学麦迪逊分校);Department of Computer Sciences Department of Biostatistics and Medical Informatics University of Wisconsin-Madison(计算机科学系生物统计学与医学信息学系威斯康星大学麦迪逊分校)

Timothy Ossowski, Danyal Maqbool, Jixuan Chen, Zefan Cai, Tyler Bradshaw, Junjie Hu

AI总结:

COMMA基准通过语言通信评估多模态多智能体系统的协作性能,揭示现有模型在智能体协作中的不足。

摘要:

多模态智能体基于大基础模型的快速发展在很大程度上忽视了其在协作任务中智能体之间基于语言的通信潜力。这种忽视在现实部署中对智能体有效性理解存在关键缺口,特别是在与人类通信时。现有智能体基准未能解决关键的智能体间通信与协作方面,特别是在智能体信息获取不平等且必须共同完成超出个体能力范围的任务时。为填补这一缺口,我们引入COMMA:一种新的谜题基准,通过语言通信评估多模态多智能体系统的协作性能。我们的基准包含多种多模态谜题,提供了一种在通信协作环境下对四种关键智能体能力进行全面评估的框架。我们的发现揭示了现有最先进模型的令人惊讶的弱点,包括强大的专有模型如GPT-4o和推理模型如o4-mini。许多链式推理模型如R1-Onevision和LLaVA-CoT在智能体间协作中难以超越随机基线,表明其在通信能力方面有增长潜力。

英文摘要:

The rapid advances of multimodal agents built on large foundation models have largely overlooked their potential for language-based communication between agents in collaborative tasks. This oversight presents a critical gap in understanding their effectiveness in real-world deployments, particularly when communicating with humans. Existing agentic benchmarks fail to address key aspects of inter-agent communication and collaboration, particularly in scenarios where agents have unequal access to information and must work together to achieve tasks beyond the scope of individual capabilities. To fill this gap, we introduce COMMA: a novel puzzle benchmark designed to evaluate the collaborative performance of multimodal multi-agent systems through language communication. Our benchmark features a variety of multimodal puzzles, providing a comprehensive evaluation across four key categories of agentic capability in a communicative collaboration setting. Our findings reveal surprising weaknesses in state-of-the-art models, including strong proprietary models like GPT-4o and reasoning models like o4-mini. Many chain of thought reasoning models such as R1-Onevision and LLaVA-CoT struggle to outperform even a random baseline in agent-agent collaboration, indicating a potential growth area in their communication abilities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑