多模态大模型

EH-Benchmark Ophthalmic Hallucination Benchmark and Agent-Driven Top-Down Traceable Reasoning Workflow

Xiaoyu Pan, Yang Bai, Ke Zou, Yang Zhou, Jun Zhou, Huazhu Fu, Yih-Chung Tham, Yong Liu

机构 * Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR)（高性能计算研究所，科技研究局（A*STAR）） ； Centre for Innovation and Precision Eye Health（创新与精准眼健康中心） ； Department of Ophthalmology, NUHS Tower Block, Level 7, 1E Kent Ridge Road, Singapore, 119228（眼科部，NUHS塔楼7层，1E Kent Ridge Road，新加坡，119228） ； Singapore Eye Research Institute, Singapore National Eye Centre, 20 College Road, Singapore, 169856（新加坡眼研究 institute，新加坡国家眼科中心，20 College Road，新加坡，169856） 纠错

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 9 figures, 5 tables. submit/6621751

UnderwaterVLA: Dual-brain Vision-Language-Action architecture for Autonomous Underwater Navigation

Zhangyuan Wang, Yunpeng Zhu, Yuqi Yan, Xiaoyuan Tian, Xinhao Shao, Meixuan Li, Weikun Li, Guangsheng Su, Weicheng Cui, Dixia Fan

机构 * School of Engineering, Westlake University（西lake大学工程学院） ； College of Information Science & Electronic Engineering, Zhejiang University（浙江大学信息科学与电子工程学院） ； College of Environmental and Resource Sciences, Zhejiang University（浙江大学环境与资源科学学院） ； Australian National University（澳大利亚国立大学） 纠错

专题命中 多模态Agent ：multimodal(abstract)；multimodal foundation model(abstract)

Comments This paper introduces the first VLA framework for AUVs, featuring a dual-brain architecture and zero-data MPC for real-world underwater navigation

JARVIS: A Neuro-Symbolic Commonsense Reasoning Framework for Conversational Embodied Agents

Kaizhi Zheng, Kaiwen Zhou, Jing Gu, Yue Fan, Jialu Wang, Zonglin Di, Xuehai He, Xin Eric Wang

机构 * University of California（加州大学） 纠错

专题命中 多模态Agent ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 19th International Conference on Neurosymbolic Learning and Reasoning

WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agent

Xinyu Geng, Peng Xia, Zhen Zhang, Xinyu Wang, Qiuchen Wang, Ruixue Ding, Chenxi Wang, Jialong Wu, Yida Zhao, Kuan Li, Yong Jiang, Pengjun Xie, Fei Huang, Jingren Zhou

专题命中 多模态Agent ：multimodal(abstract)；multi-modal(abstract)

Multi-Agent Visual-Language Reasoning for Comprehensive Highway Scene Understanding

Yunxiang Yang, Ningning Xu, Jidong J. Yang

机构 * Smart Mobility and Infrastructure Lab（智能移动与基础设施实验室） ； College of Engineering, University of Georgia（佐治亚大学工程学院） 纠错

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 16 pages, 16 figures, 8 tables

Flexible Tool Selection through Low-dimensional Attribute Alignment of Vision and Language

Guangfu Hao, Haojie Wen, Liangxuan Guo, Yang Chen, Yanchao Bi, Shan Yu

机构 * Laboratory of Brain Atlas and Brain-inspired Intelligence, Institute of Automation Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所脑图谱与类脑智能实验室） ； School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)（中国科学院大学人工智能学院） ； School of Systems Science, Beijing Normal University（北京师范大学系统科学学院） ； School of Psychological and Cognitive Sciences & Beijing Key Laboratory of Behavior and Mental Health, Peking University（北京大学心理与认知科学学院） ； IDG/McGovern Institute for Brain Research, Peking University（北京大学IDG/ McGovern脑科学研究院） ； Institute for Artificial Intelligence & Key Laboratory of Machine Perception (Ministry of Education), Peking University（北京大学人工智能研究所） ； School of Future Technology, University of Chinese Academy of Sciences (UCAS)（中国科学院大学未来技术学院） 纠错

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Medical Reasoning in the Era of LLMs: A Systematic Review of Enhancement Techniques and Applications

Wenxuan Wang, Zizhan Ma, Meidan Ding, Shiyi Zheng, Shengyuan Liu, Jie Liu, Jiaming Ji, Wenting Chen, Xiang Li, Linlin Shen, Yixuan Yuan

机构 * Renmin University of China（中国人民大学） ； The Chinese University of Hong Kong（香港中文大学） ； Shenzhen University（深圳大学） ； City University of Hong Kong（香港城市大学） ； Peking University（北京大学） ； Massachusetts General Hospital and Harvard Medical School（麻省总医院和哈佛医学院） 纠错

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Phi-Ground Tech Report: Advancing Perception in GUI Grounding

Miaosen Zhang, Ziqiang Xu, Jialiang Zhu, Qi Dai, Kai Qiu, Yifan Yang, Chong Luo, Tianyi Chen, Justin Wagle, Tim Franklin, Baining Guo

机构 * Microsoft（微软） 纠错

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

TD-Interpreter: Enhancing the Understanding of Timing Diagrams with Visual-Language Learning

Jie He, Vincent Theo Willem Kenbeek, Zhantao Yang, Meixun Qu, Ezio Bartocci, Dejan Ničković, Radu Grosu

机构 * Technische Universität Wien（维也纳技术大学） ； Shanghai Jiao Tong University（上海交通大学） ； AIT Austrian Institute of Technology（奥地利技术研究院） 纠错

专题命中 多模态Agent ：multimodal(abstract)；MLLM(abstract)

Foundation Model Driven Robotics: A Comprehensive Review

Muhammad Tayyab Khan, Ammar Waheed

机构 * School of Mechanical and Aerospace Engineering, Nanyang Technological University（南洋理工大学机械与航空航天工程学院） ； J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University（德克萨斯农工大学J. Mike Walker ’66机械工程系） 纠错

专题命中 多模态Agent ：multimodal(abstract)；cross-modal(abstract)

ControlMTR: Control-Guided Motion Transformer with Scene-Compliant Intention Points for Feasible Motion Prediction

Jiawei Sun, Chengran Yuan, Shuo Sun, Shanze Wang, Yuhang Han, Shuailei Ma, Zefan Huang, Anthony Wong, Keng Peng Tee, Marcelo H. Ang

机构 * Department of Mechanical Engineering, National University of Singapore（新加坡国立大学机械工程系） ； College of Information Science and Engineering, Northeastern University（东北大学信息科学与工程学院） ； Moovita Pte Ltd（Moovita公司） 纠错

专题命中 多模态Agent ：multimodal(abstract)；multi-modal(abstract)

Journal ref 2024 IEEE 27th International Conference on Intelligent Transportation Systems (ITSC)

Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents

Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, Yu Su

机构 * The Ohio State University（俄亥俄州立大学） ； Orby AI（Orby人工智能） 纠错

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to ICLR 2025 (Oral). Project Homepage: https://osu-nlp-group.github.io/UGround/

A Survey on (M)LLM-Based GUI Agents

Fei Tang, Haolei Xu, Hang Zhang, Siqi Chen, Xingyu Wu, Yongliang Shen, Wenqi Zhang, Guiyang Hou, Zeqi Tan, Yuchen Yan, Kaitao Song, Jian Shao, Weiming Lu, Jun Xiao, Yueting Zhuang

机构 * Zhejiang University（浙江大学） ； Microsoft Research Asia（微软亚洲研究院） 纠错

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

A Survey of LLM-based Agents in Medicine: How far are we from Baymax?

Wenxuan Wang, Zizhan Ma, Zheng Wang, Chenghan Wu, Jiaming Ji, Wenting Chen, Xiang Li, Yixuan Yuan

机构 * The Chinese University of Hong Kong（香港中文大学） ； Peking University（北京大学） ； City University of Hong Kong（香港城市大学） ； Massachusetts General Hospital and Harvard Medical School（麻省总医院和哈佛医学院） 纠错

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ACL 2025 Findings

BadNAVer: Exploring Jailbreak Attacks On Vision-and-Language Navigation

Wenqi Lyu, Zerui Li, Yanyuan Qiao, Qi Wu

机构 * Australian Institute for Machine Learning, University of Adelaide（澳大利亚机器学习研究所，阿德莱德大学） 纠错

专题命中 多模态Agent ：multimodal(abstract)；MLLM(abstract)

Comments 8 pages, 4 figures

SceneScout: Towards AI Agent-driven Access to Street View Imagery for Blind Users

Gaurav Jain, Leah Findlater, Cole Gleason

专题命中 多模态Agent ：multimodal(abstract)；MLLM(abstract)

Multi-Agent System for AI-Assisted Extraction of Narrative Arcs in TV Series

Roberto Balestri, Guglielmo Pescatore

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CL、cs.AI、cs.MM

Comments 17th International Conference on Agents and Artificial Intelligence, Porto (Portugal). 23/02/2025 - 25/02/2025

Journal ref 2025. Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 1, ISBN 978-989-758-737-5, ISSN 2184-433X, pages 663-670

GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration

Yue Fan, Handong Zhao, Ruiyi Zhang, Yu Shen, Xin Eric Wang, Gang Wu

专题命中 多模态Agent ：MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

PlanLLM: Video Procedure Planning with Refinable Large Language Models

Dejie Yang, Zijing Zhao, Yang Liu

专题命中 多模态Agent ：cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments accepted to AAAI2025

Seeing is Believing: Vision-driven Non-crash Functional Bug Detection for Mobile Apps

Zhe Liu, Cheng Li, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Yawen Wang, Jun Hu, Qing Wang

专题命中 多模态Agent ：multimodal(abstract)；MLLM(abstract)

ADAM: An Embodied Causal Agent in Open-World Environments

Shu Yu, Chaochao Lu

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Agent S: An Open Agentic Framework that Uses Computers Like a Human

Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, Xin Eric Wang

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 23 pages, 16 figures, 9 tables

"Is This It?": Towards Ecologically Valid Benchmarks for Situated Collaboration

Dan Bohus, Sean Andrist, Yuwei Bao, Eric Horvitz, Ann Paradiso

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CL、cs.AI、cs.MM

Towards Human-Level Understanding of Complex Process Engineering Schematics: A Pedagogical, Introspective Multi-Agent Framework for Open-Domain Question Answering

Sagar Srinivas Sakhinana, Geethan Sannidhi, Venkataramana Runkana

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Our paper is accepted for publication at ML4CCE workshop at ECML PKDD 2024

LEGENT: Open Platform for Embodied Agents

Zhili Cheng, Zhitong Wang, Jinyi Hu, Shengding Hu, An Liu, Yuge Tu, Pengkai Li, Lei Shi, Zhiyuan Liu, Maosong Sun

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ACL 2024 System Demonstration

OmniParser for Pure Vision Based GUI Agent

Yadong Lu, Jianwei Yang, Yelong Shen, Ahmed Awadallah

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

LLM-Optic: Unveiling the Capabilities of Large Language Models for Universal Visual Grounding

Haoyu Zhao, Wenhang Ge, Ying-cong Chen

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Project Page: https://haoyu-zhao.github.io/LLM-Optic.github.io/

Automatic Layout Planning for Visually-Rich Documents with Instruction-Following Models

Wanrong Zhu, Jennifer Healey, Ruiyi Zhang, William Yang Wang, Tong Sun

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

VideoAgent: Long-form Video Understanding with Large Language Model as Agent

Xiaohan Wang, Yuhui Zhang, Orr Zohar, Serena Yeung-Levy

专题命中 多模态Agent ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

AesopAgent: Agent-driven Evolutionary System on Story-to-Video Production

Jiuniu Wang, Zehua Du, Yuyuan Zhao, Bo Yuan, Kexiang Wang, Jian Liang, Yaxi Zhao, Yihen Lu, Gengliang Li, Junlong Gao, Xin Tu, Zhenyu Guo

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments 22 pages, 13 figures