SERPENT-VLM：使用视觉语言模型进行自精炼放射学报告生成
SERPENT-VLM : Self-Refining Radiology Report Generation Using Vision Language Models
AI总结：
提出SERPENT-VLM，通过自精炼机制和自监督损失改进放射学报告生成，减少幻觉，在IU X-ray和ROCO数据集上达到最先进性能。
摘要：
放射学报告生成（R2Gen）展示了多模态大语言模型（MLLMs）如何自动化创建准确且连贯的放射学报告。现有方法常在基于文本的报告中产生幻觉细节，这些细节不能准确反映图像内容。为缓解这一问题，我们提出了一种新颖的策略——SERPENT-VLM（使用视觉语言模型进行自精炼放射学报告生成），通过将自精炼机制集成到MLLM框架中，改进了R2Gen任务。我们采用了一种独特的自监督损失，该损失利用图像池化表示与生成的放射学文本的上下文表示之间的相似性，并结合标准因果语言建模目标，来精炼图像-文本表示。这使得模型能够通过给定图像与生成文本之间的动态交互来审查和对齐生成的文本，从而减少幻觉并持续增强细致报告生成。SERPENT-VLM在IU X-ray和放射学对象上下文（ROCO）数据集上超越了现有基线如LLaVA-Med、BiomedGPT等，实现了最先进的性能，并证明对噪声图像具有鲁棒性。一项定性案例研究强调了在R2Gen中向更复杂的MLLM框架迈出的重大进展，为医学影像领域中的自监督精炼进一步研究开辟了道路。
英文摘要：
Radiology Report Generation (R2Gen) demonstrates how Multi-modal Large Language Models (MLLMs) can automate the creation of accurate and coherent radiological reports. Existing methods often hallucinate details in text-based reports that don't accurately reflect the image content. To mitigate this, we introduce a novel strategy, SERPENT-VLM (SElf Refining Radiology RePort GENeraTion using Vision Language Models), which improves the R2Gen task by integrating a self-refining mechanism into the MLLM framework. We employ a unique self-supervised loss that leverages similarity between pooled image representations and the contextual representations of the generated radiological text, alongside the standard Causal Language Modeling objective, to refine image-text representations. This allows the model to scrutinize and align the generated text through dynamic interaction between a given image and the generated text, therefore reducing hallucination and continuously enhancing nuanced report generation. SERPENT-VLM outperforms existing baselines such as LLaVA-Med, BiomedGPT, etc., achieving SoTA performance on the IU X-ray and Radiology Objects in COntext (ROCO) datasets, and also proves to be robust against noisy images. A qualitative case study emphasizes the significant advancements towards more sophisticated MLLM frameworks for R2Gen, opening paths for further research into self-supervised refinement in the medical imaging domain.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
