arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.02483cs.CLcs.AI
收藏

MMedAgent:通过多模态智能体学习使用医疗工具

MMedAgent: Learning to Use Medical Tools with Multi-modal Agent

Stanford University(斯坦福大学);Harvard Medical School(哈佛医学院);Virginia Tech(弗吉尼亚理工大学);MSU(密歇根州立大学);CUHK(香港中文大学);Tsinghua University(清华大学);Duke University(杜克大学)

Binxu Li, Tiankai Yan, Yuanting Pan, Jie Luo, Ruiyang Ji, Jiayuan Ding, Zhe Xu, Shilong Liu, Haoyu Dong, Zihao Lin, Yixin Wang

AI总结:

针对多模态大语言模型在医疗领域通用性不足的问题,提出首个医疗专用多模态智能体MMedAgent,通过含6种工具、覆盖5模态7任务的指令微调数据集实现工具选择,性能超越现有开源方法及GPT-4o,且工具更新集成高效。

摘要:

多模态大语言模型(MLLMs)虽已取得成功,但通用性有限,且表现往往不及专用模型。近期,基于大语言模型(LLM)的智能体被开发出来以应对这些挑战,它们能根据用户输入选择合适的专用模型作为工具。然而,这类进展在医疗领域尚未得到广泛探索。为填补这一空白,本文提出了首个专为医疗领域设计的智能体,名为多模态医疗智能体(Multi-modal Medical Agent,MMedAgent)。我们整理了一个指令微调数据集,包含6种医疗工具,可解决5种模态下的7项任务,使智能体能够为给定任务选择最合适的工具。综合实验表明,与最先进的开源方法甚至闭源模型GPT-4o相比,MMedAgent在多种医疗任务上均取得了更优的性能。此外,MMedAgent在更新和集成新的医疗工具方面表现出高效性。代码和模型均已公开。

英文摘要:

Multi-Modal Large Language Models (MLLMs), despite being successful, exhibit limited generality and often fall short when compared to specialized models. Recently, LLM-based agents have been developed to address these challenges by selecting appropriate specialized models as tools based on user inputs. However, such advancements have not been extensively explored within the medical domain. To bridge this gap, this paper introduces the first agent explicitly designed for the medical field, named \textbf{M}ulti-modal \textbf{Med}ical \textbf{Agent} (MMedAgent). We curate an instruction-tuning dataset comprising six medical tools solving seven tasks across five modalities, enabling the agent to choose the most suitable tools for a given task. Comprehensive experiments demonstrate that MMedAgent achieves superior performance across a variety of medical tasks compared to state-of-the-art open-source methods and even the closed-source model, GPT-4o. Furthermore, MMedAgent exhibits efficiency in updating and integrating new medical tools. Codes and models are all available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑