arXiv 2510.086682025-11-05cs.CV

Hulu-Med：面向整体医学视觉-语言理解的透明通才模型

Hulu-Med: A Transparent Generalist Model towards Holistic Medical Vision-Language Understanding

College of Computer Science and Technology, Zhejiang University-University of Illinois Urbana-Champaign Institute（浙江大学计算机科学与技术学院）；Stomatology Hospital, School of Stomatology, Zhejiang University School of Medicine（浙江大学口腔医院）；Alibaba Inc（阿里巴巴集团）；College of Computer Science and Electronic Engineering, Hunan University（湖南大学计算机科学与电子工程学院）；Angelalign Technology Inc.（Angelalign技术有限公司）；CFAR & IHPC, Agency for Science, Technology and Research（CFAR与IHPC，新加坡科技研究局）；Department of Orthodontics, Shanghai Ninth People’s Hospital, College of Stomatology, Shanghai Jiao Tong University（上海第九人民医院正畸科，上海交通大学口腔医学院）

Songtao Jiang, Yuan Wang, Sibo Song, Tianxiang Hu, Chenyi Zhou, Bin Pu, Yan Zhang, Zhibo Yang, Yang Feng, Joey Tianyi Zhou, Jin Hao, Zijian Chen, Ruijia Wu, Tao… 展开作者

Songtao Jiang, Yuan Wang, Sibo Song, Tianxiang Hu, Chenyi Zhou, Bin Pu, Yan Zhang, Zhibo Yang, Yang Feng, Joey Tianyi Zhou, Jin Hao, Zijian Chen, Ruijia Wu, Tao Tang, Junhui Lv, Hongxia Xu, Hongwei Wang, Jun Xiao, Bin Feng, Fudong Zhu, Kenli Li, Weidi Xie, Jimeng Sun, Jian Wu, Zuozhu Liu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

Hulu-Med是一个在1670万医学多模态样本上训练的透明通才视觉-语言模型，通过医学感知令牌缩减策略统一处理文本、2D/3D图像和视频，在30个医学基准中的27个上超越现有开源模型，并提供完全可复现的训练流程。

摘要：

现实世界的临床决策需要整合异构数据，包括医学文本、2D图像、3D体数据和视频，而现有AI系统未能统一所有这些信号，限制了其实用性。本文提出Hulu-Med，一个透明的通才医学视觉-语言模型（VLM），旨在单一架构内统一纯语言、2D/3D视觉-语言和视频理解。Hulu-Med在精心构建的1670万样本语料库上训练，该语料库仅包含公开或合成数据，覆盖12个主要解剖系统和14种医学成像模态。Hulu-Med采用医学感知的令牌缩减策略，剪除冗余视觉令牌，对3D和视频输入实现高达55%的缩减，提升跨模态效率，并使其能在约4000-40000 GPU小时内完成7B-32B参数规模的训练。在30个公开的域内和域外医学基准上——涵盖文本推理、视觉问答、报告生成、多语言对话、视频理解和罕见病诊断——Hulu-Med在30个基准中的27个上超越现有开源模型，并在16个基准上优于GPT-4o等专有系统。尽管是一个VLM，Hulu-Med在纯文本HealthBench上超越GPT-4o并与GPT-o1持平。我们在社区中首次通过发布端到端的数据管理、训练流程和模型参数，提供了面向整体医学视觉-语言理解的完全透明、可复现且具有成本效益的完整流程。代码和模型可在https://github.com/ZJUI-AI4H/Hulu-Med获取。

英文摘要：

Real-world clinical decision-making requires integrating heterogeneous data, including medical text, 2D images, 3D volumes, and videos, while existing AI systems fail to unify all these signals, limiting their utility. In this paper, we introduce Hulu-Med, a transparent, generalist medical Vision-Language Model (VLM) designed to unify language-only, 2D/3D vision-language, and video understanding within a single architecture. Hulu-Med is trained on a curated corpus of 16.7 million samples, comprising exclusively public or synthetic data, spanning 12 major anatomical systems and 14 medical imaging modalities. Hulu-Med employs a medical-aware token-reduction strategy that prunes redundant visual tokens, achieving up to a 55% reduction for 3D and video inputs, improving cross-modal efficiency, and enabling training at 7B-32B parameter scales in approximately 4,000-40,000 GPU hours. Across 30 public in-domain and out-of-domain medical benchmarks-covering text reasoning, visual question answering, report generation, multilingual dialogue, video understanding, and rare disease diagnosis-Hulu-Med surpasses existing open-source models on 27 of 30 benchmarks and outperforms proprietary systems such as GPT-4o on 16 benchmarks. Despite being a VLM, Hulu-Med outperforms GPT-4o and matches GPT-o1 on the text-only HealthBench. For the first time in the community, we provide a fully transparent, reproducible and cost-effective pipeline for holistic medical vision-language understanding by releasing our end-to-end data curation, training procedures, and model parameters. Code and models are available at https://github.com/ZJUI-AI4H/Hulu-Med.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。