Dr-LLaVA：基于符号化临床基础的视觉指令微调
Dr-LLaVA: Visual Instruction Tuning with Symbolic Clinical Grounding
AI总结：
针对视觉语言模型在医学领域多轮对话中的幻觉问题，提出一种基于临床推理符号化表示的对齐算法，利用 GPT-4 生成指令微调数据并构建自动奖励函数，无需人工参与即可训练出在多轮医学对话中表现优异的 Dr-LLaVA 模型。
摘要：
视觉语言模型（VLM）可以通过分析医学图像并进行自然语言交互来辅助临床医生完成诊断和治疗任务。然而，VLM 经常表现出“幻觉”行为，生成缺乏上下文多模态信息基础的文本输出。这一挑战在医学领域尤为突出，因为我们不仅要求 VLM 的输出在单次交互中准确，还要求其在多轮对话中始终与临床推理和诊断路径保持一致。为此，我们提出了一种新的对齐算法，利用临床推理的符号化表示将 VLM 建立在医学知识的基础之上。这些表示被用于：(i) 大规模生成 GPT-4 指导的视觉指令微调数据，模拟临床医生与 VLM 的对话并展示临床推理过程；(ii) 创建一个自动奖励函数，用于在临床医生与 VLM 交互期间评估 VLM 生成内容的临床有效性。我们的算法消除了在训练数据生成或奖励模型构建中对人工参与的需求，与标准的基于人类反馈的强化学习（RLHF）相比降低了成本。我们应用该对齐算法开发了 Dr-LLaVA，这是一个经过微调的对话式 VLM，专门用于分析骨髓病理切片，在多轮医学对话中展现出强大的性能。
英文摘要：
Vision-Language Models (VLM) can support clinicians by analyzing medical images and engaging in natural language interactions to assist in diagnostic and treatment tasks. However, VLMs often exhibit "hallucinogenic" behavior, generating textual outputs not grounded in contextual multimodal information. This challenge is particularly pronounced in the medical domain, where we do not only require VLM outputs to be accurate in single interactions but also to be consistent with clinical reasoning and diagnostic pathways throughout multi-turn conversations. For this purpose, we propose a new alignment algorithm that uses symbolic representations of clinical reasoning to ground VLMs in medical knowledge. These representations are utilized to (i) generate GPT-4-guided visual instruction tuning data at scale, simulating clinician-VLM conversations with demonstrations of clinical reasoning, and (ii) create an automatic reward function that evaluates the clinical validity of VLM generations throughout clinician-VLM interactions. Our algorithm eliminates the need for human involvement in training data generation or reward model construction, reducing costs compared to standard reinforcement learning with human feedback (RLHF). We apply our alignment algorithm to develop Dr-LLaVA, a conversational VLM finetuned for analyzing bone marrow pathology slides, demonstrating strong performance in multi-turn medical conversations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
