arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.12184cs.LGcs.AI
收藏

医疗领域多模态大模型的民主化:TinyLLaVA-Med 在资源受限环境中的高效医疗诊断

Democratizing MLLMs in Healthcare: TinyLLaVA-Med for Efficient Healthcare Diagnostics in Resource-Constrained Settings

New York University Abu Dhabi(纽约大学阿布扎比分校)

Aya El Mir, Lukelo Thadei Luoga, Boyuan Chen, Muhammad Abdullah Hanif, Muhammad Shafique

AI总结:

针对资源受限医疗环境,通过指令微调 TinyLLaVA 得到 TinyLLaVA-Med,在低功耗(18.9W)下实现接近最先进水平的医疗视觉问答准确率。

摘要:

在医疗领域部署多模态大语言模型(MLLMs)受到其高计算需求和显著内存需求的阻碍,这对于像 Nvidia Jetson Xavier 这样的资源受限设备尤其具有挑战性。这一问题在需要先进诊断但资源有限的偏远医疗环境中尤为明显。在本文中,我们提出了一种针对通用多模态大语言模型 TinyLLaVA 的优化方法,我们对其进行了调整并重新命名为 TinyLLaVA-Med。这一调整涉及受 LLaVA-Med 训练流程启发,在医学数据集上对 TinyLLaVA 进行指令微调和微调。我们的方法成功降低了计算复杂度和功耗,TinyLLaVA-Med 运行功耗为 18.9W,使用 11.9GB 内存,同时在 VQA-RAD 上达到 64.54% 的准确率,在 SLAKE 上针对封闭式问题达到 70.70% 的准确率。因此,TinyLLaVA-Med 在硬件受限、计算资源低的环境中实现了部署可行性,同时保持了基本功能,并提供了接近最先进模型的准确率。

英文摘要:

Deploying Multi-Modal Large Language Models (MLLMs) in healthcare is hindered by their high computational demands and significant memory requirements, which are particularly challenging for resource-constrained devices like the Nvidia Jetson Xavier. This problem is particularly evident in remote medical settings where advanced diagnostics are needed but resources are limited. In this paper, we introduce an optimization method for the general-purpose MLLM, TinyLLaVA, which we have adapted and renamed TinyLLaVA-Med. This adaptation involves instruction-tuning and fine-tuning TinyLLaVA on a medical dataset by drawing inspiration from the LLaVA-Med training pipeline. Our approach successfully minimizes computational complexity and power consumption, with TinyLLaVA-Med operating at 18.9W and using 11.9GB of memory, while achieving accuracies of 64.54% on VQA-RAD and 70.70% on SLAKE for closed-ended questions. Therefore, TinyLLaVA-Med achieves deployment viability in hardware-constrained environments with low computational resources, maintaining essential functionalities and delivering accuracies close to state-of-the-art models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑