D-Rax：利用多模态数据和专家模型预测的领域专用放射学助手
D-Rax: Domain-specific Radiologic assistant leveraging multi-modal data and eXpert model predictions
AI总结：
本文提出领域专用放射学助手D-Rax，通过微调LLaVA-Med并结合MIMIC-CXR数据、VQA对及多专家模型预测，提升胸部X光对话分析的准确性，以辅助临床诊断。
摘要：
大型视觉语言模型（VLMs）从研究到通用场景的适用性取得了令人难以置信的进展。LLaVA-Med作为生物医学领域开创性的大型语言与视觉助手，能够执行多模态生物医学图像和数据分析，为放射科医生提供自然语言界面。尽管它具有高度泛化能力并支持多模态数据，但目前仍受限于大型语言模型领域中众所周知的挑战。响应中的幻觉和不精确可能导致误诊，这目前阻碍了VLM的临床适应性。为了在医疗保健中创建精确、用户友好的模型，我们提出了D-Rax——一种领域专用、对话式的放射学辅助工具，可用于获取关于特定放射学图像的见解。在本研究中，我们增强了胸部X光（CXR）图像的对话式分析，以支持放射学报告，提供来自医学影像的全面见解，并帮助制定准确的诊断。D-Rax是通过在我们精心构建的增强指令遵循数据上微调LLaVA-Med架构实现的，这些数据包括图像、指令，以及从MIMIC-CXR影像数据、CXR相关视觉问答（VQA）对和多个专家AI模型的预测结果中得出的疾病诊断和人口统计学预测。我们在开放和封闭式对话的评估中观察到响应在统计上显著的改进。利用最先进的诊断模型与VLM相结合的力量，D-Rax使临床医生能够使用自然语言与医学图像进行交互，这有望简化他们的决策过程、提高诊断准确性并节省时间。
英文摘要：
Large vision language models (VLMs) have progressed incredibly from research to applicability for general-purpose use cases. LLaVA-Med, a pioneering large language and vision assistant for biomedicine, can perform multi-modal biomedical image and data analysis to provide a natural language interface for radiologists. While it is highly generalizable and works with multi-modal data, it is currently limited by well-known challenges that exist in the large language model space. Hallucinations and imprecision in responses can lead to misdiagnosis which currently hinder the clinical adaptability of VLMs. To create precise, user-friendly models in healthcare, we propose D-Rax -- a domain-specific, conversational, radiologic assistance tool that can be used to gain insights about a particular radiologic image. In this study, we enhance the conversational analysis of chest X-ray (CXR) images to support radiological reporting, offering comprehensive insights from medical imaging and aiding in the formulation of accurate diagnosis. D-Rax is achieved by fine-tuning the LLaVA-Med architecture on our curated enhanced instruction-following data, comprising of images, instructions, as well as disease diagnosis and demographic predictions derived from MIMIC-CXR imaging data, CXR-related visual question answer (VQA) pairs, and predictive outcomes from multiple expert AI models. We observe statistically significant improvement in responses when evaluated for both open and close-ended conversations. Leveraging the power of state-of-the-art diagnostic models combined with VLMs, D-Rax empowers clinicians to interact with medical images using natural language, which could potentially streamline their decision-making process, enhance diagnostic accuracy, and conserve their time.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
