COMMA: A Communicative Multimodal Multi-Agent Benchmark
Department of Computer Sciences University of Wisconsin-Madison(计算机科学系威斯康星大学麦迪逊分校);Department of Computer Sciences UC San Diego(计算机科学系加州大学圣地亚哥分校);Department of Radiology University of Wisconsin-Madison(放射学系威斯康星大学麦迪逊分校);Department of Computer Sciences Department of Biostatistics and Medical Informatics University of Wisconsin-Madison(计算机科学系生物统计学与医学信息学系威斯康星大学麦迪逊分校)
The rapid advances of multimodal agents built on large foundation models have largely overlooked their potential for language-based communication between agents in collaborative tasks. This oversight presents a critical gap in understanding their effectiveness in real-world deployments, particularly when communicating with humans. Existing agentic benchmarks fail to address key aspects of inter-agent communication and collaboration, particularly in scenarios where agents have unequal access to information and must work together to achieve tasks beyond the scope of individual capabilities. To fill this gap, we introduce COMMA: a novel puzzle benchmark designed to evaluate the collaborative performance of multimodal multi-agent systems through language communication. Our benchmark features a variety of multimodal puzzles, providing a comprehensive evaluation across four key categories of agentic capability in a communicative collaboration setting. Our findings reveal surprising weaknesses in state-of-the-art models, including strong proprietary models like GPT-4o and reasoning models like o4-mini. Many chain of thought reasoning models such as R1-Onevision and LLaVA-CoT struggle to outperform even a random baseline in agent-agent collaboration, indicating a potential growth area in their communication abilities.
Voice-Interactive Surgical Agent for Multimodal Patient Data Control
Image Guided Precision Cancer Surgery Institute, College of Medicine, Korea University(影像引导精准癌症手术研究所,韩国大学医学院);Department of Radiology, Massachusetts General Hospital(放射科,麻省总医院);Department of Thoracic and Cardiovascular Surgery, Korea University Guro Hospital, College of Medicine, Korea University(胸外科和心血管外科,韩国大学Guro医院,韩国大学医学院);Department of Biomedical Sciences, College of Medicine, Korea University(生物医学科学系,韩国大学医学院)
AgentHijack: Visual Patch Attacks on Multimodal Computer-Use Agents
Hainan University(海南大学);University of Chinese Academy of Sciences National Computer Network Intrusion Protection Center(中国科学院大学国家计算机网络入侵防护中心);Lancaster University(兰卡斯特大学)
See-Control: A Multimodal Agent Framework for Smartphone Interaction with a Robotic Arm
University College London(伦敦大学学院);Imperial College London(伦敦帝国学院);Huawei Noah’s Ark Lab(华为诺亚实验室);ShanghaiTech University(上海科技大学);Southern University of Science(南方科技大学)
SafeMobile: Chain-level Jailbreak Detection and Automated Evaluation for Multimodal Mobile Agents
Nanyang Technological University(南洋理工大学);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);Beihang University(北京航空航天大学);Sun Yat-sen University(中山大学);Beijing Institute of Technology(北京理工大学);National University of Singapore(新加坡国立大学)
Enhancing Instruction-Following Capability of Visual-Language Models by Reducing Image Redundancy
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Kuaishou Technology(快手科技);CAIR, HKISI, Chinese Academy of Sciences(中国科学院香港创新研究院人工智能与机器人创新中心)
A Survey on Agentic Multimodal Large Language Models
Nanyang Technological University, Singapore(南洋理工大学,新加坡);Chinese University of Hong Kong, Shenzhen, China(香港中文大学(深圳));Shenzhen Research Institute of Big Data, China(大数据研究 institute);Sun Yat-sen University, China(中山大学);City University of Hong Kong, China(香港城市大学);Communication University of China, China(通信大学)