多模态大语言模型(MLLMs)虽已取得成功,但通用性有限,且表现往往不及专用模型。近期,基于大语言模型(LLM)的智能体被开发出来以应对这些挑战,它们能根据用户输入选择合适的专用模型作为工具。然而,这类进展在医疗领域尚未得到广泛探索。为填补这一空白,本文提出了首个专为医疗领域设计的智能体,名为多模态医疗智能体(Multi-modal Medical Agent,MMedAgent)。我们整理了一个指令微调数据集,包含6种医疗工具,可解决5种模态下的7项任务,使智能体能够为给定任务选择最合适的工具。综合实验表明,与最先进的开源方法甚至闭源模型GPT-4o相比,MMedAgent在多种医疗任务上均取得了更优的性能。此外,MMedAgent在更新和集成新的医疗工具方面表现出高效性。代码和模型均已公开。
英文摘要:
Multi-Modal Large Language Models (MLLMs), despite being successful, exhibit limited generality and often fall short when compared to specialized models. Recently, LLM-based agents have been developed to address these challenges by selecting appropriate specialized models as tools based on user inputs. However, such advancements have not been extensively explored within the medical domain. To bridge this gap, this paper introduces the first agent explicitly designed for the medical field, named \textbf{M}ulti-modal \textbf{Med}ical \textbf{Agent} (MMedAgent). We curate an instruction-tuning dataset comprising six medical tools solving seven tasks across five modalities, enabling the agent to choose the most suitable tools for a given task. Comprehensive experiments demonstrate that MMedAgent achieves superior performance across a variety of medical tasks compared to state-of-the-art open-source methods and even the closed-source model, GPT-4o. Furthermore, MMedAgent exhibits efficiency in updating and integrating new medical tools. Codes and models are all available.
Overview of the NLPCC 2025 Shared Task 4: Multi-modal, Multilingual, and Multi-hop Medical Instructional Video Question Answering Challenge
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(深圳先进技术研究所,中国科学院);School of Computer Science and Technology, Beijing Institute of Technology(计算机科学与技术学院,北京理工大学);School of Engineering, Westlake University(工程学院,西湖大学)
M$^3$-Med: A Benchmark for Multi-lingual, Multi-modal, and Multi-hop Reasoning in Medical Instructional Video Understanding
Beijing Institute of Technology(北京理工大学);The Hong Kong Polytechnic University(香港理工大学);Chinese Academy of Sciences(中国科学院);Air Force Hospital of Southern Theater Command of PLA(中国人民解放军南部战区空军医院);Shenzhen Traditional Chinese Medicine Hospital(深圳中医药医院)
A Spectrum Evaluation Benchmark for Medical Multi-Modal Large Language Models
The City University of Hong Kong(香港城市大学);The Chinese University of Hong Kong(香港中文大学);Shenzhen University(深圳大学);National Yang Ming Chiao Tung University(国立阳明交通大学);Taipei Veterans General Hospital(台北荣民总医院);Stanford University(斯坦福大学)
Eye-gaze Guided Multi-modal Alignment for Medical Representation Learning
School of Automation, Northwestern Polytechnical University(西北工业大学自动化学院);City University of Hong Kong(香港城市大学);The University of Texas at Arlington(德克萨斯大学阿灵顿分校);School of Biomedical Engineering, ShanghaiTech University(上海科技大学生物医学工程学院);Shanghai United Imaging Intelligence Co., Ltd.(上海联影智能医疗科技有限公司);Shanghai Clinical Research and Trial Center(上海临床研究与试验中心);University of Georgia(佐治亚大学);Massachusetts General Hospital(麻省总医院)
Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Training
Shenzhen Research Institute of Big Data(深圳大数据研究院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Sun Yat-sen University(中山大学);Pazhou Lab(琶洲实验室)