多模态大模型

Process-Supervised Reinforcement Learning for Interactive Multimodal Tool-Use Agents

Weiting Tan, Xinghua Qu, Ming Tu, Meng Ge, Andy T. Liu, Philipp Koehn, Lu Lu

机构 * Johns Hopkins University（约翰霍普金斯大学） 纠错

专题命中 多模态Agent ：multimodal(title)；multi-modal(abstract)；分类 cs.CL、cs.AI

Project Riley: Multimodal Multi-Agent LLM Collaboration with Emotional Reasoning and Voting

Ana Rita Ortigoso, Gabriel Vieira, Daniel Fuentes, Luis Frazão, Nuno Costa, António Pereira

机构 * Polytechnic University of Leiria（莱里亚理工大学） 纠错

专题命中 多模态Agent ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments 28 pages, 5 figures. Submitted for review to Information Fusion

Multimodal Masked Autoencoder Pre-training for 3D MRI-Based Brain Tumor Analysis with Missing Modalities

Lucas Robinet, Ahmad Berjaoui, Elizabeth Cohen-Jonathan Moyal

机构 * Oncopole（奥恩波勒） ； IRT Saint Exupéry（伊尔特圣埃克苏佩里研究所） ； INSERM Cancer Research Center of Toulouse（图卢兹癌症研究中心） 纠错

专题命中 多模态Agent ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Griffon v2: Advancing Multimodal Perception with High-Resolution Scaling and Visual-Language Co-Referring

Yufei Zhan, Shurong Zheng, Yousong Zhu, Hongyin Zhao, Fan Yang, Ming Tang, Jinqiao Wang

机构 * Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所基础模型研究中心） ； School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院） ； Peng Cheng Laboratory, Shenzhen, China（鹏城实验室） ； Wuhan AI Research, Wuhan, China（武汉人工智能研究所） 纠错

专题命中 多模态Agent ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted by ICCV 2025. Codes and datasets are released at https://github.com/jefferyZhan/Griffon

WSI-Agents: A Collaborative Multi-Agent System for Multi-Modal Whole Slide Image Analysis

Xinheng Lyu, Yuci Liang, Wenting Chen, Meidan Ding, Jiaqi Yang, Guolin Huang, Daokun Zhang, Xiangjian He, Linlin Shen

机构 * College of Artificial Intelligence, Shenzhen University, China（深圳大学人工智能学院） ； School of Computer Science, University of Nottingham Ningbo China, China（诺丁汉宁波大学计算机学院） ； College of Computer Science and Software Engineering, Shenzhen University, China（深圳大学计算机科学与软件工程学院） ； Department of Electrical Engineering, City University of Hong Kong, Hong Kong（香港城市大学电气工程系） ； Wuyi University, China（武夷大学） 纠错

专题命中 多模态Agent ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Multimedia Verification Through Multi-Agent Deep Research Multimodal Large Language Models

Huy Hoan Le, Van Sy Thinh Nguyen, Thi Le Chi Dang, Vo Thanh Khang Nguyen, Truong Thanh Hung Nguyen, Hung Cao

机构 * University of New Brunswick（新 Brunswick大学） ； Institute for Clarity in Documentation（文档清晰研究所） ； Inria Paris-Rocquencourt（巴黎-罗克琴特研究所） ； Rajiv Gandhi University（拉贾·甘地大学） ； Tsinghua University（清华大学） ； Palmer Research Laboratories（帕勒尔研究实验室） 纠错

专题命中 多模态Agent ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments 33rd ACM International Conference on Multimedia (MM'25) Grand Challenge on Multimedia Verification

Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agents

Tianyi Men, Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao

机构 * The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences（认知与决策智能复杂系统重点实验室，自动化研究所，中国科学院） ； School of Artificial Intelligence, University of Chinese Academy of Sciences（人工智能学院，中国科学院大学） 纠错

专题命中 多模态Agent ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments ACL 2025 Main

GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior

Penghao Wu, Shengnan Ma, Bo Wang, Jiaheng Yu, Lewei Lu, Ziwei Liu

机构 * S-Lab, Nanyang Technological University（南洋理工大学S实验室） ； SenseTime Research（商汤科技研究院） 纠错

专题命中 多模态Agent ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments Project Page at https://penghao-wu.github.io/GUI_Reflection/

UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents

Han Xiao, Guozhi Wang, Yuxiang Chai, Zimu Lu, Weifeng Lin, Hao He, Lue Fan, Liuyang Bian, Rui Hu, Liang Liu, Shuai Ren, Yafei Wen, Xiaoxin Chen, Aojun Zhou, Hongsheng Li

机构 * CUHK MMLab（香港中文大学MML实验室） ； vivo AI Lab（vivo人工智能实验室） ； CPII under InnoHK（创新科技署下的CPII） 纠错

专题命中 多模态Agent ：MLLM(title)；image-text(abstract)；分类 cs.CV、cs.CL

Comments https://github.com/Euphoria16/UI-Genie

Development and Validation of Engagement and Rapport Scales for Evaluating User Experience in Multimodal Dialogue Systems

Fuma Kurata, Mao Saeki, Masaki Eguchi, Shungo Suzuki, Hiroaki Takatsu, Yoichi Matsuyama

机构 * Waseda University, Tokyo, Japan（大阪大学） 纠错

专题命中 多模态Agent ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Journal ref Proceedings of the 14th International Workshop on Spoken Dialogue Systems Technology, Hokkaido, Japan, 2024

ColonScopeX: Leveraging Explainable Expert Systems with Multimodal Data for Improved Early Diagnosis of Colorectal Cancer

Natalia Sikora, Robert L. Manschke, Alethea M. Tang, Peter Dunstan, Dean A. Harris, Su Yang

专题命中 多模态Agent ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments Published to AAAI-25 Bridge Program

EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agents

Zhili Cheng, Yuge Tu, Ran Li, Shiqi Dai, Jinyi Hu, Shengding Hu, Jiahao Li, Yang Shi, Tianyu Yu, Weize Chen, Lei Shi, Maosong Sun

专题命中 多模态Agent ：multimodal(title,abstract)；分类 cs.CV、cs.CL

TopV-Nav: Unlocking the Top-View Spatial Reasoning Potential of MLLM for Zero-shot Object Navigation

Linqing Zhong, Chen Gao, Zihan Ding, Yue Liao, Huimin Ma, Shifeng Zhang, Xu Zhou, Si Liu

专题命中 多模态Agent ：MLLM(title,abstract)；分类 cs.CV、cs.AI

Comments 10 pages

Multimodal Transformer Models for Turn-taking Prediction: Effects on Conversational Dynamics of Human-Agent Interaction during Cooperative Gameplay

Young-Ho Bae, Casey C. Bennett

专题命中 多模态Agent ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments 36 pages

BuildingView: Constructing Urban Building Exteriors Databases with Street View Imagery and Multimodal Large Language Mode

Zongrong Li, Yunlei Su, Hongrong Wang, Wufan Zhao

专题命中 多模态Agent ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments 15 pages, 6 figures

Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage

Zhi Gao, Bofei Zhang, Pengxiang Li, Xiaojian Ma, Tao Yuan, Yue Fan, Yuwei Wu, Yunde Jia, Song-Chun Zhu, Qing Li

专题命中 多模态Agent ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Comments ICLR 2025, https://mat-agent.github.io/

PlotGen: Multi-Agent LLM-based Scientific Data Visualization via Multimodal Feedback

Kanika Goswami, Puneet Mathur, Ryan Rossi, Franck Dernoncourt

专题命中 多模态Agent ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Multimodal MRI-Ultrasound AI for Prostate Cancer Detection Outperforms Radiologist MRI Interpretation: A Multi-Center Study

Hassan Jahanandish, Shengtian Sang, Cynthia Xinran Li, Sulaiman Vesal, Indrani Bhattacharya, Jeong Hoon Lee, Richard Fan, Geoffrey A. Sonna, Mirabela Rusu

专题命中 多模态Agent ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Evolver: Chain-of-Evolution Prompting to Boost Large Multimodal Models for Hateful Meme Detection

Jinfa Huang, Jinsheng Pan, Zhongwei Wan, Hanjia Lyu, Jiebo Luo

专题命中 多模态Agent ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments accepted by COLING 2025

A Multimodal Social Agent

Athina Bikaki, Ioannis A. Kakadiaris

专题命中 多模态Agent ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments 9 pages

AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations

Gaurav Verma, Rachneet Kaur, Nishan Srishankar, Zhen Zeng, Tucker Balch, Manuela Veloso

专题命中 多模态Agent ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments 18 pages, 3 figures, an abridged version to appear in NeurIPS 2024 AFM Workshop

OpenOmni: A Collaborative Open Source Tool for Building Future-Ready Multimodal Conversational Agents

Qiang Sun, Yuanyi Luo, Sirui Li, Wenxiao Zhang, Wei Liu

专题命中 多模态Agent ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments Published in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP 2024) Best Demo Paper Award at EMNLP 2024

Journal ref EMNLP 2024 (System Demonstrations), pp. 46-52

Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models

Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, Ranjay Krishna

专题命中 多模态Agent ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments Accepted to NeurIPS 2024. Project and codes url: https://visualsketchpad.github.io/

GenRL: Multimodal-foundation world models for generalization in embodied agents

Pietro Mazzaglia, Tim Verbelen, Bart Dhoedt, Aaron Courville, Sai Rajeswar

专题命中 多模态Agent ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments Presented at NeurIPS 2024

OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization

Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Hongming Zhang, Tianqing Fang, Zhenzhong Lan, Dong Yu

专题命中 多模态Agent ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks

Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, Liqiang Nie

专题命中 多模态Agent ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted by NeurIPS 2024

SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?

John Yang, Carlos E. Jimenez, Alex L. Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R. Narasimhan, Diyi Yang, Sida I. Wang, Ofir Press

专题命中 多模态Agent ：multimodal(title,abstract)；分类 cs.CL、cs.AI

MMedAgent: Learning to Use Medical Tools with Multi-modal Agent

Binxu Li, Tiankai Yan, Yuanting Pan, Jie Luo, Ruiyang Ji, Jiayuan Ding, Zhe Xu, Shilong Liu, Haoyu Dong, Zihao Lin, Yixin Wang

专题命中 多模态Agent ：multi-modal(title,abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2024

Music2P: A Multi-Modal AI-Driven Tool for Simplifying Album Cover Design

Joong Ho Choi, Geonyeong Choi, Ji-Eun Han, Wonjin Yang, Zhi-Qi Cheng

专题命中 多模态Agent ：multi-modal(title,abstract)；分类 cs.AI、cs.MM

Comments Accepted at CIKM 2024 Demo Paper track. Project available at https://github.com/JC-78/Music2P

Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?

Ruisheng Cao, Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yeqiao Fu, Hongcheng Gao, Xinzhuang Xiong, Hanchong Zhang, Yuchen Mao, Wenjing Hu, Tianbao Xie, Hongshen Xu, Danyang Zhang, Sida Wang, Ruoxi Sun, Pengcheng Yin, Caiming Xiong, Ansong Ni, Qian Liu, Victor Zhong, Lu Chen, Kai Yu, Tao Yu

专题命中 多模态Agent ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments 34 pages, 14 figures, 10 tables