arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.08002cs.CLcs.CV
收藏

迈向临床可及的放射学基础模型：开放访问与轻量化及自动化评估

Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluation

Microsoft Research（微软研究院）；University of Washington（华盛顿大学）；Stanford University（斯坦福大学）；University of Southern California（南加州大学）；University of California, Davis（加利福尼亚大学戴维斯分校）
Juan Manuel Zambrano Chaves, Shih-Cheng Huang, Yanbo Xu, Hanwen Xu, Naoto Usuyama, Sheng Zhang, Fei Wang, Yujia Xie, Mahmoud Khademi, Ziyi Yang, Hany Awadalla, … 展开作者

Juan Manuel Zambrano Chaves, Shih-Cheng Huang, Yanbo Xu, Hanwen Xu, Naoto Usuyama, Sheng Zhang, Fei Wang, Yujia Xie, Mahmoud Khademi, Ziyi Yang, Hany Awadalla, Julia Gong, Houdong Hu, Jianwei Yang, Chunyuan Li, Jianfeng Gao, Yu Gu, Cliff Wong, Mu Wei, Tristan Naumann, Muhao Chen, Matthew P. Lungren, Akshay Chaudhari, Serena Yeung-Levy, Curtis P. Langlotz, Sheng Wang, Hoifung Poon

发表信息：
Nature Communications volume 16, Article number: 3108 (2025)
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究提出开源轻量级多模态模型LlaVA-Rad（7B），通过模块化训练轻量级适配器结合预训练模型，在放射学报告生成和跨模态检索任务上超越GPT-4V等大模型，并引入CheXprompt自动化评估指标，实现单GPU私有环境下的临床可及性。

摘要：

大型基础模型的缩放定律和卓越性能推动了此类模型在生物医学中的开发与利用。然而，尽管在一些生物医学基准上取得了初步可喜的成果，但在这些模型应用于真实世界临床之前，仍有重大挑战需要解决。诸如GPT-4V等前沿通用领域模型在多模态生物医学应用中仍存在显著的性能差距。更重要的是，较少被关注的实际问题，包括可及性、模型成本和繁琐的手工评估，使得临床医生难以直接在私有患者数据上使用最先进的大型模型。在此，我们探索训练开源小型多模态模型（SMMs），以弥合放射学中未满足临床需求的能力差距。为最大化数据效率，我们采用模块化方法，结合图像和文本模态的最先进预训练模型，并专注于训练轻量级适配器（adapter）将每种模态映射到文本嵌入空间，以LLaVA-Med为例。在训练方面，我们汇集了超过69.7万个放射学图像-文本对的大型数据集。在评估方面，我们提出CheXprompt，一种基于GPT-4的事实性评估指标，并证明其与专家评估的一致性。为探索最佳实践，我们对数据工程和多模态训练中的各种选择进行了系统消融研究。最终得到的LlaVA-Rad（7B）模型在报告生成和跨模态检索等标准放射学任务上取得最先进结果，甚至超越GPT-4V和Med-PaLM M（84B）等更大模型。LlaVA-Rad推理快速，可在私有环境中的单块V100 GPU上运行，为真实世界临床应用提供有前景的最先进工具。

英文摘要：

The scaling laws and extraordinary performance of large foundation models motivate the development and utilization of such models in biomedicine. However, despite early promising results on some biomedical benchmarks, there are still major challenges that need to be addressed before these models can be used in real-world clinics. Frontier general-domain models such as GPT-4V still have significant performance gaps in multimodal biomedical applications. More importantly, less-acknowledged pragmatic issues, including accessibility, model cost, and tedious manual evaluation make it hard for clinicians to use state-of-the-art large models directly on private patient data. Here, we explore training open-source small multimodal models (SMMs) to bridge competency gaps for unmet clinical needs in radiology. To maximize data efficiency, we adopt a modular approach by incorporating state-of-the-art pre-trained models for image and text modalities, and focusing on training a lightweight adapter to ground each modality to the text embedding space, as exemplified by LLaVA-Med. For training, we assemble a large dataset of over 697 thousand radiology image-text pairs. For evaluation, we propose CheXprompt, a GPT-4-based metric for factuality evaluation, and demonstrate its parity with expert evaluation. For best practice, we conduct a systematic ablation study on various choices in data engineering and multimodal training. The resulting LlaVA-Rad (7B) model attains state-of-the-art results on standard radiology tasks such as report generation and cross-modal retrieval, even outperforming much larger models such as GPT-4V and Med-PaLM M (84B). The inference of LlaVA-Rad is fast and can be performed on a single V100 GPU in private settings, offering a promising state-of-the-art tool for real-world clinical applications.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑